• All Files

Google BigQuery Reader

Licensing options for this format begin with FME Professional Edition.

The BigQuery Reader module enables FME to read BigQuery attribute data.

Overview

BigQuery is a managed NoSQL database service offered by Google. The BigQuery reader communicates using HTTP requests managed by the Google APIs Client Library for Java.

This chapter assumes familiarity with BigQuery.

For more information, please visit the BigQuery home at

https://cloud.google.com/products/bigquery/

Usage Notes

The performance of this format is dependent on the amount of memory allocated to the Java Virtual Machine (JVM). The following environment variables allow you to specify memory available to Java Plugins:

  • FME_JVM_MIN_HEAP_SIZE: Initial heap size for initializing the JVM. If unset, the default value is 1024K.

  • FME32_JVM_MIN_HEAP_SIZE: Same as FME_JVM_MIN_HEAP_SIZE, but applies to 32-bit FME Desktop on Windows, and if set, takes precedence over FME_JVM_MIN_HEAP_SIZE.

These variables must be set in multiples of 1024 and greater than 1 MB. To indicate kilobytes, megabytes, or gigabytes, append k or K, m or M, or g or G respectively. For example, any of these values is acceptable:

6291456

6144k

6m

  • FME_JVM_MAX_HEAP_SIZE: Maximum heap size for initializing the JVM. If unset, the default value is 16384K.

  • FME32_JVM_MAX_HEAP_SIZE: Same as FME_JVM_MAX_HEAP_SIZE, but applies to 32-bit FME Desktop on Windows, and if set, takes precedence over FME_JVM_MAX_HEAP_SIZE.

These variables must be set in multiples of 1024 and greater than 2 MB. To indicate kilobytes, megabytes, or gigabytes, append k or K, m or M, or g or G respectively. For example, any of these values is acceptable:

83886080

81920k

80m

Note: To pass additional parameters used by FME to the Java Virtual Machine, use the JAVA_TOOL_OPTIONS environment variable.

Reader Overview

When reading from BigQuery, each table is considered a feature type in FME and each row of a table is one feature in FME as long as there are no repeated fields.

In general, large volumes of data may not be read from BigQuery. A WHERE clause may be specified to limit the amount of data read. Alternatively, choosing a destination table and enabling “Allow Large Results” will allow large volumes of data to be read.

Open the Workbench Google BigQuery Reader: Feature Type Parameters to view the table list options.