Delta Lake Reader Parameters

Database formats include a Database Connection parameter that defines and stores authentication information. For general information about sharing database connections, please see Using Database Connections.

Note that Database Connection parameters differ slightly, depending on context and/or database format.

Connection

From the Connection parameter in a database format, you can do one of the following:

  • Select an existing, previously defined connection. See the section Reusing a Database Connection in Using Database Connections
  • Select Add Database Connection to define a new connection. See database-specific parameters below, as well as the section Adding a Database Connection in a Workspace in Using Database Connections. The new connection can be made visible only to the current user, or can be shared among multiple users.

Database Connection

Table Name Mapping

As part of the Delta Lake connection, you must specify the URLs of the delta tables you wish to access, along with a name for each table. These names can be used to conveniently refer to delta table URLs when selecting which tables to read in the reader, and when writing SQL queries against a Delta Lake connection in the DatabaseExecutor.

Cloud Provider

As part of the Delta Lake connection, you must specify the cloud storage platform where the delta tables are hosted. Each choice requires additional configuration for authenticated access. The currently supported platforms are:

Amazon Web Services

AWS supports 3 credential types:

  1. AWS SDK Default Authentication – The AWS CLI default credential precedence (described here) will be used.

  2. SSO – The AWS CLI single-sign-on credentials are used. You must run aws sso login using the AWS CLI.

  3. Secret Key – An AWS IAM secret access key is used.

All credential types require the AWS region of the S3 bucket containing the delta table to be specified (e.g., us-west-1).

Azure

Azure supports 4 credential types:

  1. Azure SDK Default Authentication – Azure’s default authentication flow will be used.

  2. SSO – The Azure CLI single-sign-on credentials are used. You must run az login using the Azure CLI.

  3. Account Key – Specify the Azure storage account name and secret key.

  4. Connection String – Any Azure storage connection string can be used. Useful for high configuration.

Google Cloud Storage

GCS supports authentication via Access Key and Secret obtained from the Cloud Storage dashboard settings, under the Interoperability tab.

Constraints

After specifying the database connection, click the Browse button (...) to select tables for import. A connection window appears while the system retrieves the tables from the database.

Once the Select Tables dialog appears, you can select one or more tables. Click OK to dismiss the window and add the selected table name(s) to the Tables parameter.

Schema Attributes

Use this parameter to expose Format Attributes in FME Workbench when you create a workspace:

  • In a dynamic scenario, it means these attributes can be passed to the output dataset at runtime.
  • In a non-dynamic scenario, this parameter allows you to expose additional attributes on multiple feature types. Click the browse button to view the available format attributes (which are different for each format) for the reader.

Spatial

Coordinate systems may be extracted from input feature data sources, may come predefined with FME, or may be user-defined. FME allows different output and input coordinate systems, and performs the required coordinate conversions when necessary.

How FME Processes Coordinate Systems in the Workspace

If a coordinate system is specified in both the source format and the workspace, the coordinate system in the workspace is used. The coordinate system specified in the source format is not used, and a warning is logged. If a source coordinate system is not specified in the workspace and the format or system does not store coordinate system information, then the coordinate system is not set for the features that are read.

If a destination coordinate system is set and the feature has been tagged with a coordinate system, then a coordinate system conversion is performed to put the feature into the destination system. This happens right before the feature enters into the writer.

If the destination coordinate system was not set, then the features are written out in their original coordinate system.

If a destination coordinate system is set, but the source coordinate system was not specified in the workspace or stored in the source format, then no conversion is performed. The features are simply tagged with the output system name before being written to the output dataset.

For systems that know their coordinate system, the Coordinate System field will display Read from Source and FME will read the coordinate system from the source dataset. For most other input sources, the field will display Unknown (which simply means that FME will use default values). In most cases, the default value is all you'll need to perform the translation.

You can always choose to override the defaults and choose a new coordinate system. Select More Coordinate Systems from the drop-down menu to open the Coordinate System Gallery.

Changing a Reprojection

To perform a reprojection, FME typically uses the CS-MAP reprojection engine, which includes definitions for thousands of coordinate systems, with a large variety of projections, datums, ellipsoids, and units. However, GIS applications have slightly different algorithms for reprojecting data between different coordinate systems. To ensure that the data FME writes matches exactly to your existing data, you can use the reprojection engine from a different application.

Note  To use this feature, you must have a supported GIS application installed on your system (for example, Esri ArcGIS®).

To change the reprojection engine, Select Workspace Parameters > Spatial > Reprojection Engine. In the example shown, you can select Esri (but the selection here depends on your installed applications):

For more information, see the links below: