AzureBlobStorageConnector

Connects to Azure Blob Storage to manage files and folders.

Typical Uses

Managing datasets on Azure Blob Storage by uploading, downloading, and deleting files and folders
Transferring file contents into or out of an attribute in FME
Retrieving file and folder names, paths, links and other information from Azure Blob Storage to use elsewhere in a workspace

How does it work?

The AzureBlobStorageConnector uses your Azure Storage account credentials to perform various management tasks on Azure Blob Storage:

Action	Task
Delete	Delete files and folders.
Download	Download a file or folder to an attribute, file, or folder.
List	List the item details for the contents of a folder.
Upload	Upload a file, folder, or the contents of an attribute.

Once valid account credentials have been provided, you may browse to available content for the selected Action.

Usage Notes

This transformer cannot move or copy files between different locations. Consider using multiple connectors for these tasks, by downloading files/folders and re-uploading them to a new location.
The FeatureReader can access Azure Blob Storage directly but does not download a local copy of the dataset.

Configuration

Input Ports

Output Ports

Output

Features with added attributes, as specified in parameters and according to Action:

Action	Output
Delete	Input feature unchanged.
Download	Input feature with details about the downloaded item.
List	Multiple features, one for each item in the specified location, with details about the items.
Upload	Input feature with details about the uploaded item.

Parameters

Authentication

Credential Source

Select the type of credentials to use:

Web Connection	(Recommended) Use an Azure Storage Web Connection.
SAS	Use a Shared Account Signature.
Embedded	Embed an account name and access key as transformer parameters.
Default Azure Credential	Obtain credentials from Microsoft Entra ID, Managed Identity, or Azure CLI.
Environment Variables	Use credentials already configured on the system in the environment variables AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_ACCESS_KEY.
Anonymous	Access public Blob Storage containers without authenticating.

Account

When Credential Source is Web Connection, select or create a Web Connection connecting to a Microsoft Azure Storage Web Service.

See Using Web Connections.

Account Name

When Credential Source is SAS, Embedded, Default Azure Credential, or Anonymous, specify the account name.

Account Key

When Credential Source is Embedded, specify the account key.

Custom Domain

(Optional) If the blob storage service is not located at <account>.blob.core.windows.net, specify a location to override the host.

Shared Access Signature (SAS)

When Credential Source is SAS, provide an SAS.

Action > Download

Download Options

Container

Specify the container the file or folder is in.

Path

Specify the full path of the file or folder on Azure Blob Storage to download.

Advanced

(Optional) To download only a portion of the blob, specify a byte range:

The starting and ending values are inclusive.
If no End of Byte Range is given, all bytes after the Start of Byte Range will be downloaded.
If End of Byte Range is given, Start of Byte Range must be provided.

Example:

To download the first 512 bytes of the blob:

Start of Byte Range =0

End of Byte Range=511

Start of Byte Range	Specify the start of the byte range to use for downloading a section of the blob.
End of Byte Range	End of byte range to use for downloading a section of the blob.

Destination

Download as	Select a destination type for downloaded items: Attribute: The downloaded object will be stored in the specified attribute. File: The downloaded file will be stored in the specified folder. Folder: The downloaded folder object will be stored in the specified folder.
Download to Attribute	When Download as is Attribute, specify the attribute to contain the contents of the downloaded item. Encoding FME will attempt to convert the attribute's contents to a UTF-8 string. If this fails, the attribute's contents will be created as raw binary. To ensure that the original bytes are always preserved and never converted to UTF-8, use Download as >File instead.
Response Body Encoding	When Download as is Attribute, select an encoding method. Options include various character set encodings which ensure the output is in a human-readable form. If Auto Detect from HTTP Header is specified but there is no encoding in the Content-Type metadata associated with the data, the output will be returned as raw bytes.
Download to Folder	When Download as is File or Folder, specify the path to the folder that will store the downloaded item.
If File Exists	When Download as is File or Folder, select a method for handling existing files: Reject: Leave the existing file intact and output the feature via the <Rejected> port. Overwrite: Overwrite the existing file.
Include Subfolders	When Download as is Folder, select a method for handling subfolders: Yes: Include all subfolders No: Do not include subfolders.

Output Attributes

Attributes to Add

Select any available attributes to be included on the output features:

_download_path	Path of the downloaded object on the local file system. This attribute will not be added for Download as Attribute.
_full_path	Full path of the file or folder.
_name	Name of the file or folder.
_url	URL to the file or folder.

Action > List

List Options

Container

Specify the container the folder is in.

Path

Specify the full path of the folder on Azure Blob Storage to list.

Include Subfolders

Select a method for handling subfolders:

Yes: Include all subfolders
No: Do not include subfolders.

Output Attributes

Attributes to Add

Select any available attributes to be included on the output features:

_created	Date and time when the file was created
_file_or_folder	Whether the item is a file or a folder
_file_size	File size (not present for folders)
_full_path	Full path of the file or folder.
_last_modified	Date and time when the file was last modified
_lease_duration	The lease type (fixed or infinite). Possible values: fixed, infinite.
_lease_state	The current state of the blob's lease. Possible values: available, leased, expired, breaking, broken.
_lease_status	Indicates if the blob is locked (leased) or unlocked. Possible values: locked, unlocked.
_name	Name of the file or folder.
_relative_path	Path to the item relative to the list path
_url	URL to the file or folder.

Action > Upload

Data Source

Upload	Select the source type: Attribute: Upload the contents of an attribute. File: Upload a file. Folder: Upload a folder and its contents. Note that when working with large objects, File is generally the better choice as the data will be read from disk and not need to be held in memory.
File to Upload	When Upload is File, specify the file to be uploaded.
Folder to Upload	When Upload is Folder, specify the folder to be uploaded.
Include Subfolders	When Upload is Folder, select a method for handling subfolders: Yes: Upload all subfolders No: Do not include subfolders.
Contents Only	When Upload is Folder, select a method for uploading the content of folders: Yes: Upload only the contents of the folder to the destination. No: Upload the folder and its contents to the destination.
Attribute to Upload as File	When Upload is Attribute, specify the attribute containing the content for upload.

Upload Options

Container	Specify the container to upload to. If it does not exist, a new container will be created.
Path	Specify the full path of the folder to upload to.
Upload with File Name	When Upload is Attribute, specify the filename, including extension, to contain the uploaded content.
If File Exists	Select a method of handling conflicts with existing files at the upload location: Create New Version: The new file is uploaded as a new version of the existing file. Reject: If uploading a file, the input feature will be rejected. Overwrite: Any existing files or folders will be overwritten by the uploaded items.
Maximum Block Size (MB)	Specify the size of the blocks to create when uploading (provided the file is larger than 4MB). The maximum block size Azure Blob Storage supports is 100MB. For more information about block size, see: Understanding block blobs, append blobs, and page blobs.

Metadata

This table specifies metadata items to associate with each uploaded file.

Each entry consists of a Name and Value.

Name

Select an available name from the list or enter a name directly:

Cache-Control
Content-Disposition
Content-Encoding
Content-Language
Content-MD5
Content-Type
x-ms-meta-

If the name matches a recognized HTTP header value, it will be set such that HTTP requests for the uploaded file will return those headers.

Note that:

If Content-Type is not specified, then it will be set based on the extension of the uploaded file if possible.
If Content-Encoding is unset, a value will also be determined if applicable.

Value

Specify a value.

Generate Shared Access Signature

When Upload is File or Attribute, a Shared Access Signature (SAS) can be generated for the uploaded file.

When enabled, the attributes _sas_url and _sas_token will be added to the output feature.

SAS generation requires authentication with an Account Key.

For more information, see: Grant limited access to Azure Storage resources using shared access signatures (SAS).

Permissions	Select the desired permission level: Read Read/Write Read/Write/Delete
SAS Token Expiry Time	Specify when the token should expire, in datetime format.
IP Address or Range	(Optional) If desired, specify an IP address or range of IP addresses, limiting access to this range only. Range format is: 203.0.113.0–203.0.113.255
Restrict to HTTPS	Select an option: Yes (default): The token can only be used via HTTPS. No: No restrictions.

Output Attributes

Attributes to Add

Select any available attributes to be included on the output features:

_full_path	Full path of the file or folder.
_name	Name of the file or folder.
_url	URL to the file or folder.

Editing Transformer Parameters

Transformer parameters can be set by directly entering values, using expressions, or referencing other elements in the workspace such as attribute values or user parameters. Various editors and context menus are available to assist. To see what is available, click beside the applicable parameter.

How to Set Parameter Values

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters.

Using the Text Editor

The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.

Text Editor

Using the Arithmetic Editor

The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.

Arithmetic Editor

Conditional Values

Set values depending on one or more test conditions that either pass or fail.

Parameter Condition Definition Dialog

Content

Expressions and strings can include a number of functions, characters, parameters, and more.

When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.

Content Types

String Functions	These functions manipulate and format strings.
Special Characters	A set of control characters is available in the Text Editor.
Math Functions	Math functions are available in both editors.
Date/Time Functions	Date and time functions are available in the Text Editor.
Math Operators	These operators are available in the Arithmetic Editor.
FME Feature Functions	These return primarily feature-specific values.
FME Parameters	FME and workspace-specific parameters may be used.
Creating and Modifying User Parameters	Create your own editable parameters.

Dialog Options - Tables

Table Tools

Transformers with table-style parameters have additional tools for populating and manipulating values.

Row Reordering

Enabled once you have clicked on a row item. Choices include:

Add a row
Remove a row
Move current row up one
Move current row down one
Move current row to top
Move current row to bottom

Cut, Copy, and Paste

Enabled once you have clicked on a row item. Choices include:

Cut a row - delete and copy to clipboard
Copy a row to the clipboard
Paste a row from the clipboard

Cut, copy, and paste may be used within a transformer, or between transformers.

Filter

Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output.

Import

Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers.

Reset/Refresh

Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers.

Note: Not all tools are available in all transformers.

For more information, see Transformer Parameter Menu Options.

Reference

Processing Behavior	Feature-Based
Feature Holding	No
Dependencies	Azure Storage Account
Aliases	MicrosoftAzureBlobStorageConnector
History	Released FME 2019.0

FME Community

The FME Community has a wealth of FME knowledge with over 20,000 active members worldwide. Get help with FME, share knowledge, and connect with users globally.

Search for all results about the AzureBlobStorageConnector on the FME Community.

Examples may contain information licensed under the Open Government Licence – Vancouver, Open Government Licence - British Columbia, and/or Open Government Licence – Canada.

AzureBlobStorageConnector

Typical Uses

How does it work?

Usage Notes

Configuration

Input Ports

Output Ports

Parameters

Delete Options

Output Attributes

Download Options

Destination

Encoding

Output Attributes

List Options

Output Attributes

Data Source

Upload Options

Metadata

Generate Shared Access Signature

Output Attributes

Editing Transformer Parameters

Defining Values

Using the Text Editor

Using the Arithmetic Editor

Conditional Values

Content

Table Tools

Reference

FME Community