FME Transformers: 2025.2 BETA

Categories
Integrations

Integrations

Web

Web

Workflows

Workflows

Related Transformers
AzureBlobStorageConnector
AzureFileStorageConnector
BoxConnector
CesiumIonConnector
ComprehendConnector
DropboxConnector
GoogleCloudStorageConnector
GoogleDriveConnector
HDFSConnector
OneDriveConnector
RekognitionConnector
SQSConnector
Categories
Integrations

Integrations

Web

Web

Workflows

Workflows

Related Transformers
AzureBlobStorageConnector
AzureFileStorageConnector
BoxConnector
CesiumIonConnector
ComprehendConnector
DropboxConnector
GoogleCloudStorageConnector
GoogleDriveConnector
HDFSConnector
OneDriveConnector
RekognitionConnector
SQSConnector

S3Connector

Connects to Amazon S3 to manage files and folders.

Jump to Configuration

Typical Uses

  • Managing datasets on S3 by uploading, downloading, and deleting files and folders
  • Transferring file contents into or out of an attribute in FME
  • Retrieving file and folder names, paths, links and other information from S3 to use elsewhere in a workspace

How does it work?

The S3Connector uses your Amazon S3 account credentials to perform various management tasks:

Action Task

Delete

Delete files and folders.

Download

Download a file or folder to an attribute, file, or folder.

List

List the item details for the contents of a folder.

Upload

Upload a file, folder, or the contents of an attribute.

Once valid account credentials have been provided, you may browse to available content for the selected Action.

Usage Notes

  • This transformer cannot move or copy files between different locations. Consider using multiple connectors for these tasks, by downloading files/folders and re-uploading them to a new location.

  • The FeatureReader can access S3 directly but does not download a local copy of the dataset.

Configuration

Input Ports

This transformer accepts any feature.

Output Ports

Features with added attributes, as specified in parameters and according to Action:

Action Output

Delete

Input feature unchanged.

Download

Input feature with details about the downloaded item.

List

Multiple features, one for each item in the specified location, with details about the items.

Upload

Input feature with details about the uploaded item.

Input features are output here unmodified, in addition to any other output locations (Output or <Rejected>).

Features that cause the operation to fail are output through this port. An fme_rejection_code attribute describing the category of the error will be added, along with a more descriptive fme_rejection_message which contains more specific details as to the reason for the failure.

If an Input feature already has a value for fme_rejection_code, this value will be removed.

Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.

Parameters

Credential Source

Select the type of credentials to use:

  • Anonymous: Access public S3 buckets without authenticating.
  • Web Connection (Recommended): Use an Amazon Web Services web connection.
  • Embedded: Embed an access key ID and secret access key as parameters in the transformer.
  • System: Use credentials already configured on the system. This allows for using IAM role-based authentication. For more information about configuring AWS credentials system-wide, see Getting started with the AWS CLI.

Account

When Credential Source is Web Connection, select or create a Web Connection connecting to an Amazon S3 Compatible Storage or Amazon Web Services Web Service.

See Using Web Connections.

Region

 

Select the AWS Region in which the specified Bucket resides.

If the default, US East, N. Virginia (us-east-1), is specified, and the Bucket does not reside there, the operation will still succeed, but specifying the correct region is recommended.

Alternate Endpoint

(Optional) Specify an alternate S3-compatible API.

These are typically object storage services from providers other than AWS, or on-premises solutions.

Verify HTTPS Certificates

Select a verification option:

  • Yes (default): Ensure that HTTPS certificates are signed by a trusted authority.
  • No: Do not check the authenticity of HTTPS certificates. Note that the connection is still encrypted, but another party can more easily intercept the transmitted data.

AWS CA Bundle

(Optional) Provide an AWS CA Bundle specifying the path to a certificate bundle to use for certificate validation.

If set, this has precedence over the value of the AWS_CA_BUNDLE environment variable.

Requester Pays

Select a payment option to use if required:

  • Yes: Allow AWS to charge the account associated with the provided credentials to be charged.
  • No (Default): Do not charge the requester.

Embedded Credentials

When Credential Source is Embedded:

Access Key ID

Provide S3 account credentials.

Secret Access Key

Provide S3 account credentials.

Session Token

(Optional) Use temporary security credentials to connect to S3.

Action

Select an operation to perform. Choices include:

  • Delete
  • Download
  • List
  • Upload

Delete Options

Bucket

Specify the bucket the file or folder is in.

Path

Specify the full path of a file or folder on S3 to delete.

Output Attributes

No attributes are added for this Action.

Download Options

Bucket

Specify the bucket the file or folder is in.

Path

Specify the full path of a file or folder on S3 to download.

S3 Acceleration

Select an option:

  • Yes: Allow Amazon S3 Transfer Acceleration on the bucket, if enabled.
  • No (Default): Do not use.

When allowing S3 Acceleration, keep in mind the following:

  • The specified Bucket name must be DNS-compliant.
  • Your Amazon account is charged for performing accelerated requests.

See Enabling and using S3 Transfer Acceleration.

Destination

Download as

Select a destination type for downloaded items:

  • Attribute: The downloaded object will be stored in the specified attribute.
  • File: The downloaded file will be stored in the specified folder.
  • Folder: The downloaded folder object will be stored in the specified folder.

Download to Attribute

When Download as is Attribute, specify the attribute to contain the contents of the downloaded item.

Encoding

FME will attempt to convert the attribute's contents to a UTF-8 string. If this fails, the attribute's contents will be created as raw binary. To ensure that the original bytes are always preserved and never converted to UTF-8, use Download as >File instead.

Response Body Encoding

When Download as is Attribute, select an encoding method.

Options include various character set encodings which ensure the output is in a human-readable form.

If Auto Detect from HTTP Header is specified but there is no encoding in the Content-Type metadata associated with the data, the output will be returned as raw bytes.

Download to Folder

When Download as is File or Folder, specify the path to the folder that will store the downloaded item.

If File Exists

When Download as is File or Folder, select a method for handling existing files:

  • Reject: Leave the existing file intact and output the feature via the <Rejected> port.
  • Overwrite: Overwrite the existing file.

Include Subfolders

When Download as is Folder, select a method for handling subfolders:

  • Yes: Include all subfolders
  • No: Do not include subfolders.

Output Attributes

Attributes to Add

Select any available attributes to be included on the output features:

_download_path

Path of the downloaded object on the local file system. This attribute will not be added for Download as Attribute.

List Options

Bucket

Specify the bucket the folder is in.

Path

Specify the full path to the folder on S3 to list.

Include Subfolders

Select a method for handling subfolders:

  • Yes: Include all subfolders
  • No: Do not include subfolders.

Output Attributes

Attributes to Add

Select any available attributes to be included on the output features:

_bucket_name

Name of the bucket to which the file or folder was uploaded or ID of an object on S3.

_file_last_modified

Last modified date of an object on S3.

_file_or_folder

Type (file or folder) of an object on S3.

_file_size

Size of a file object on S3.

_full_path

Full path to an object on S3

_name

Name of an object on S3.

_relative_path

Relative path to an object on S3.

_url

URL of an object on S3.

Data Source

Upload

Select the source type:

  • Attribute: Upload the contents of an attribute.
  • File: Upload a file.
  • Folder: Upload a folder and its contents.

Note that when working with large objects, File is generally the better choice as the data will be read from disk and not need to be held in memory.

S3 treats file uploads of the same name, in a specific folder, as duplicates, but does not allow multiple instances in a specific folder. You must upload the file to a different folder than the existing one or the original will be overwritten.

File to Upload

When Upload is File, specify the file to be uploaded.

Folder to Upload

When Upload is Folder, specify the folder to be uploaded.

Include Subfolders

When Upload is Folder, select a method for handling subfolders:

  • Yes: Upload all subfolders
  • No: Do not include subfolders.

Contents Only

When Upload is Folder, select a method for handling subfolders:

  • Yes: Upload all subfolders
  • No: Do not include subfolders.

Attribute to Upload as File

When Upload is Attribute, specify the attribute containing the content for upload.

Upload Options

Bucket

Specify the bucket the folder is in.

Path

Specify the full path to the folder on S3 to upload to.

Upload with File Name

When Upload is Attribute, specify the filename, including extension, to contain the uploaded content.

If File Exists

Select a method of handling conflicts with existing files at the upload location:

  • Create New Version: The new file is uploaded as a new version of the existing file.
  • Reject: If uploading a file, the input feature will be rejected.
  • Overwrite: Any existing files or folders will be overwritten by the uploaded items.

S3 Acceleration

Select an option:

  • Yes: Allow Amazon S3 Transfer Acceleration on the bucket, if enabled.
  • No (Default): Do not use.

When allowing S3 Acceleration, keep in mind the following:

  • The specified Bucket name must be DNS-compliant.
  • Your Amazon account is charged for performing accelerated requests.

See Enabling and using S3 Transfer Acceleration.

Encryption

Select an encryption option:

  • Server-Side Encryption (AES-256): Instructs Amazon S3 to encrypt the data after it is uploaded using the 256-bit Advanced Encryption Standard (AES). For more information, see this AWS Blog post.
  • None: Data is not encrypted after upload.

Canned ACL

Select a pre-defined option for permissions for each uploaded object:

  • None
  • Private
  • Public Read
  • Public Read/Write
  • Authenticated Read
  • Bucket Owner Read
  • Bucket Owner Full Control

For more information, see Access control list (ACL) overview.

Generate Presigned URL

If enabled, a presigned URL is generated for the uploaded file, and stored in the _presigned_url attribute. This allows access to the file without credentials for the bucket. For more information, see https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html.

Generate Presigned URL

When enabled, a presigned URL is generated for the uploaded object and stored in the _presigned_url attribute.

This allows access to the file without credentials for the bucket. For more information, see Sharing objects with presigned URLs.

Expiry

(Optional) Specify a number of seconds for which the link will remain valid.

Metadata

This table specifies metadata items to associate with each uploaded file, specifying header fields in the HTTP upload request.

Each entry consists of a Name and Value.

Name

Select an available name from the list or enter a name directly.

Options include metadata and ACL permissions parameters.

Custom metadata fields will automatically be prefixed with x-amz-meta- by AWS.

Value

Specify a value.

Output Attributes

Attributes to Add

Select any available attributes to be included on the output features:

_bucket_name

Name of the bucket to which the file or folder was uploaded or ID of an object on S3.

_full_path

Full path to an object on S3

_name

Name of an object on S3.

_url

URL of an object on S3.

Editing Transformer Parameters

Transformer parameters can be set by directly entering values, using expressions, or referencing other elements in the workspace such as attribute values or user parameters. Various editors and context menus are available to assist. To see what is available, click beside the applicable parameter.

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters.

Using the Text Editor

The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.

Text Editor

Using the Arithmetic Editor

The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.

Arithmetic Editor

Conditional Values

Set values depending on one or more test conditions that either pass or fail.

Parameter Condition Definition Dialog

Content

Expressions and strings can include a number of functions, characters, parameters, and more.

When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.

String Functions

These functions manipulate and format strings.

Special Characters

 A set of control characters is available in the Text Editor.

Math Functions

Math functions are available in both editors.
Date/Time Functions Date and time functions are available in the Text Editor.

Math Operators

These operators are available in the Arithmetic Editor.

FME Feature Functions

These return primarily feature-specific values.

FME Parameters

FME and workspace-specific parameters may be used.
Creating and Modifying User Parameters Create your own editable parameters.

Table Tools

Transformers with table-style parameters have additional tools for populating and manipulating values.

Row Reordering

Enabled once you have clicked on a row item. Choices include:

  • Add a row
  • Remove a row
  • Move current row up one
  • Move current row down one
  • Move current row to top
  • Move current row to bottom

Cut, Copy, and Paste

Enabled once you have clicked on a row item. Choices include:

  • Cut a row - delete and copy to clipboard
  • Copy a row to the clipboard
  • Paste a row from the clipboard

Cut, copy, and paste may be used within a transformer, or between transformers.

Filter

Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output.

Import

Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers.

Reset/Refresh

Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers.

Note: Not all tools are available in all transformers.

For more information, see Transformer Parameter Menu Options.

Reference

Processing Behavior

Feature-Based

Feature Holding

No
Dependencies S3 account
Aliases S3Deleter S3Downloader S3ObjectLister S3Uploader
History Released FME 2019.0

FME Online Resources

The FME Community and Support Center Knowledge Base have a wealth of information, including active forums with 35,000+ members and thousands of articles.

Search for all results about the S3Connector on the FME Community.

 

Examples may contain information licensed under the Open Government Licence – Vancouver, Open Government Licence - British Columbia, and/or Open Government Licence – Canada.