HDFSConnector

Accesses an HDFS (Hadoop Distributed File System) to upload, download, or delete files and folders; or list the contents of a folder from an HDFS service.

Jump to Configuration

Typical Uses

  • Manage datasets on HDFS by uploading, downloading, and deleting files and folders
  • Transfer a file's contents (such as XML, point cloud, or raster) into or out of an attribute in FME
  • Read downloaded HDFS data using the FeatureReader, or upload data written by the FeatureWriter to HDFS
  • Retrieve file and folder names, paths, links and other information from HDFS to use elsewhere in a workspace.

How does it work?

The HDFSConnector uses your HDFS account credentials (either via a previously defined FME web connection, or by setting up a new FME web connection right from the transformer) to access the file storage service.

Depending on your choice of actions, it will upload or download files, folders, and attributes; list information from the service; or delete items from the service. On uploads, path attributes are added to the output features. On List actions, file/folder information is added as attributes.

Examples

Usage Notes

  • This transformer cannot be used to directly move or copy files between different HDFS locations. However, multiple HDFSConnectors can be used to accomplish these tasks.
  • The FeatureReader can access HDFS directly (without using the HDFSConnector), however, a local copy of the dataset will not be created.

Configuration

Input Ports

Output Ports

Parameters

The remaining parameters available depend on the value of the Request > HDFS Action parameter. Parameters for each HDFS Action are detailed below.

Editing Transformer Parameters

Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.

Dialog Options - Tables

Transformers with table-style parameters have additional tools for populating and manipulating values.

Reference

Processing Behavior

Feature-Based

Feature Holding

No

Dependencies HDFS account
FME Licensing Level FME Base Edition and above
Aliases  
History Released FME 2018.0

FME Community

The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.

Search for all results about the HDFSConnector on the FME Community.