Amazon DynamoDB Reader/Writer

Licensing options for this format begin with FME Desktop Professional Edition.

The DynamoDB Reader/Writer module enables FME to read and write DynamoDB attribute data.

Overview

DynamoDB is a managed NoSQL database service offered by Amazon Web Services.

The DynamoDB reader/writer communicates using HTTP requests managed by the Amazon Web Services Java SDK.

This chapter assumes familiarity with DynamoDB, the attribute types supported, and its indexing mechanisms.

For more information, please visit the DynamoDB home at

http://aws.amazon.com/dynamodb/

Usage Notes

The performance of this format is dependent on the amount of memory allocated to the Java Virtual Machine (JVM). The following environment variables allow you to specify memory available to Java Plugins:

  • FME_JVM_MIN_HEAP_SIZE: Initial heap size for initializing the JVM. If unset, the default value is 1024K.

  • FME32_JVM_MIN_HEAP_SIZE: Same as FME_JVM_MIN_HEAP_SIZE, but applies to 32-bit FME Desktop on Windows, and if set, takes precedence over FME_JVM_MIN_HEAP_SIZE.

These variables must be set in multiples of 1024 and greater than 1 MB. To indicate kilobytes, megabytes, or gigabytes, append k or K, m or M, or g or G respectively. For example, any of these values is acceptable:

6291456

6144k

6m

  • FME_JVM_MAX_HEAP_SIZE: Maximum heap size for initializing the JVM. If unset, the default value is 16384K.

  • FME32_JVM_MAX_HEAP_SIZE: Same as FME_JVM_MAX_HEAP_SIZE, but applies to 32-bit FME Desktop on Windows, and if set, takes precedence over FME_JVM_MAX_HEAP_SIZE.

These variables must be set in multiples of 1024 and greater than 2 MB. To indicate kilobytes, megabytes, or gigabytes, append k or K, m or M, or g or G respectively. For example, any of these values is acceptable:

83886080

81920k

80m

Note: To pass additional parameters used by FME to the Java Virtual Machine, use the JAVA_TOOL_OPTIONS environment variable.

Reader Overview

FME considers a DynamoDB dataset to be a database containing a collection of tables. The tables must be defined in Workbench before they can be read.

When reading from the DynamoDB database, each table is considered a feature type in FME and each row of a table one feature in FME.

Open the Workbench DynamoDB Reader Parameters to view the table list options.

Writer Overview

The DynamoDB writer module stores attributes into a DynamoDB database. The DynamoDB writer provides the following capabilities:

  • Index Creation: The DynamoDB writer can set up and populate indexes as part of the loading process. When creating a table, a partition key index must be specified. One additional column can be indexed using DynamoDB’s sort key index. The DynamoDB writer also supports the creation of a single global or local secondary index.
  • Batch Writes: The DynamoDB writer writes in batches of up to 25 items in order to ensure a faster data loading process. The maximum number of items processed in a single batch can be changed using the Batch Size writer parameter.