ArcGISOnlineConnector

Categories

Integrations

Web

Workflows

Related Transformers

AzureBlobStorageConnector

GoogleCloudStorageConnector

S3Connector

Accesses Esri ArcGIS Online or Portal for ArcGIS to upload, manage, update, download, delete, publish, share, or list information about items from an Esri ArcGIS account.

Jump to Configuration

Typical Uses

  • Manage datasets and supplementary files on ArcGIS Online or Portal for ArcGIS by uploading, downloading, publishing, sharing, and deleting files and folders
  • Transfer a file's contents into or out of an attribute in FME
  • Read downloaded ArcGIS data using the FeatureReader, or upload data written by the FeatureWriter to ArcGIS Online or Portal for ArcGIS
  • Retrieve file and folder names, paths, links and other information from ArcGIS Online or Portal for ArcGIS for use elsewhere in a workspace

How does it work?

The ArcGISOnlineConnector uses either your ArcGIS Online or Portal for ArcGIS account credentials (either via a previously defined FME web connection, or by setting up a new FME web connection right from the transformer) to access the file storage service.

Depending on your choice of actions, it will upload or download files, folders, and attributes; list information from the service; share items which already exist in the user’s online content; publish items as feature layers; or delete items from the service. On Uploads, link attributes are added to the output features. On List actions, links and file/folder information are added as attributes.

Usage Notes

  • This transformer cannot be used to directly move or copy files between different ArcGIS user content folders or locations. However, multiple ArcGISOnlineConnectors can be used to accomplish these tasks.
  • Uploading data to ArcGIS is performed via synchronous multipart upload. Each file chunk is 20MB. The connector will upload these chunks consecutively and if a chunk upload fails, the connector will retry each upload up to five times.
  • Early termination of translations when performing publish/update/upload actions may result in malformed ArcGIS item entries. (When publishing, a feature service may be created but any title/tags specified for it may not be honored). (When updating/uploading, it is possible for the metadata to be uploaded before the file content has been uploaded).
  • The FeatureReader can access online ArcGIS content directly (without using the ArcGISOnlineConnector), however, a local copy of the dataset will not be created.

Configuration

Input Ports

This transformer accepts any feature.

Output Ports

The output of this transformer will vary depending on the Action performed.

  • A Delete action outputs no additional attributes.
  • A Download action can save to a file, a folder, or an attribute. If it saves to a file or folder, it will output the path it saved to an attribute.
  • A List action will output a new feature for each file/folder found in the path specified. Each of these new features will have attributes listing various pieces of information about the object.
  • A Publish action will output information about the feature service it created, including its id, feature service type and URLs to the feature service itself as well as its ArcGIS content page.
  • A Share action outputs no additional attributes.
  • An Upload action will output the id of the uploaded item and a link to its ArcGIS content page.
  • An Update action will output the id of the updated item and a link to its ArcGIS content page.

The incoming feature is output through this port.

Features that cause the operation to fail are output through this port. An fme_rejection_code attribute describing the category of the error will be added, along with a more descriptive fme_rejection_message attribute which contains more specific details as to the reason for the failure.

Note: If a feature comes in to the ArcGISOnlineConnector already having a value for fme_rejection_code, this value will be removed.

Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.

Parameters

ArcGIS Connection

Performing operations against an ArcGIS Online or Portal for ArcGIS account requires the use of OAuth2 authentication using a Web Connection.

To create an ArcGIS connection, click the 'ArcGIS Connection' drop-down box and select 'Add Web Connection...'. The connection can then be managed via Tools -> FME Options... -> Web Connections.

To create an ArcGIS Portal Connection, users must create their own web connection based on the Esri ArcGIS Portal Web Service template provided with FME. From FME Workbench, open FME Options > Web Connections and click the Manage Services button. Navigate to the Esri ArcGIS Portal template. The resulting web connection can be used with the transformer.

Action

The type of operation to perform. Choices include:

  • Delete
  • Download
  • List
  • Publish
  • Share
  • Update
  • Upload

The remaining parameters available depend on the value of the Request > Action parameter. Parameters for each Action are detailed below.

Delete an existing item/folder from a user’s own ArcGIS Online content.

Content Source

Path

The Item ID of a file or folder on ArcGIS Online or Portal for ArcGIS to delete. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. The user must be the owner of this item.

Notes:

  • The root folder of a user’s content cannot be specified for deletion.
  • If a folder is specified which contains a delete protected item, all items in the folder which are not delete protected will still be deleted, but the folder and the deleted protected item(s) will remain and the input feature will be rejected
  • If an invalid item/folder id is specified, an API error will be logged, but the input feature will not be rejected

Download a file/folder to an attribute/file/folder from a user’s ArcGIS content or a group they have access to.

Content Source

Content Source

Select the source type of the content to be downloaded

  • User: The user’s own ArcGIS content
  • Group: A group the user is a member of or a group in their organization that they have access to

Group ID

The ArcGIS group which the content is linked to.

Note:

  • To download all of the content from a group, choose to Download as Folder and set the path to ‘/’

Path

The ArcGIS Item ID of a file or folder to download. If valid credentials have been provided as a web connection, you may browse to a location. Depending on the value of the Content Source parameter, this will list either the contents of the specified group or the user’s own ArcGIS content. Only items which can be downloaded will be displayed. If the user has permission to access items stored in other locations, the id of those items can be manually entered in this field.

Destination

Download as

Select whether to store the downloaded data in a File, Folder or Attribute.

  • File: The downloaded object will be stored in the specified folder. The filename used will be the ArcGIS title of the object and the original extension stored with the filename. If the ArcGIS title contained any file-unsafe characters, these will be replaced with underscores.
  • Folder: The downloaded folder object will be created and stored inside the specified folder. If the content type was set to group, the created folder will be named after the group. Otherwise, it will have the same name as the target folder. If the root folder was specified for download, the contents of the root folder will not be nested inside a content folder - they will be downloaded directly to the specified folder
  • Attribute: The downloaded object will be stored in the specified attribute. Note that this will always assume system encoding. Use the AttributeEncoder if your contents need to be a specific encoding type.

Download to Attribute

Specify the attribute that will store the contents of the downloaded file. Valid for Download as Attribute only.

Note: FME will attempt to convert the attribute's contents to a UTF-8 string. If this fails, the attribute's contents will be created as raw binary. To ensure that the original bytes are always preserved and never converted to UTF-8, use Download as File instead.

Download to Folder

Specify the path to the folder that will store the downloaded file. Valid for Download as File or Folder.

If File Exists

Whether to overwrite existing files in the target destination. Valid for Download as File or Folder.

Include Subfolders

Whether to include subfolders when downloading a folder. Due to the way that ArcGIS content is structured, only the root folder contains subfolders. Valid for Download as Folder .

Output Attributes

_download_path

Return the path the file/folder was downloaded to. If a folder was downloaded, the path will have a trailing slash.

List the item details for the contents of a group, a user’s folder, a user’s root folder, or all of a user’s content.

Content Source

Content Source

Select the source type of the content to be listed

  • User: The user’s own ArcGIS content
  • Group: A group the user is a member of or a group in their organization that they have access to

Group ID

The ArcGIS group which the content is linked to.

User Content Folder

The ArcGIS folder ID to list the contents of, or ‘/’ for root. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only folders may be selected. Valid when Content Source is set to User.

Include Subfolders

Whether to include subfolder content when listing. Due to the way that ArcGIS content is structured, only the root folder contains subfolders. Valid when Content Source is set to User.

Output Attributes

_id

Return the ID of a stored ArcGIS object.

_name

Return the title of a stored ArcGIS object.

_file_or_folder

Return the type (file or folder) of a stored ArcGIS object.

_private_url

Return the link to an object on ArcGIS. If the object is an item, this will direct the user to the ArcGIS web content page for their item. If the object is a folder, this will direct the user to the ArcGIS web content listing of items for their folder.

_created

Return the creation time (UNIX timestamp) of a stored ArcGIS object.

_file_size

Return the size of a stored ArcGIS object (in bytes). File only.

_file_type

Return the file extension of a stored ArcGIS object. (for example, ‘zip’ or ‘csv’) File only, and only if the file is downloadable.

_is_downloadable

Returns a flag (Yes/No) depending on whether or not the ArcGISOnlineConnector can be used to download this object. File only.

_item_type

Return the ArcGIS type of a stored ArcGIS object. File only.

_json_response

Return the JSON response provided directly from the API for this item.

_last_modified

Return the last modified time (UNIX timestamp) of a stored ArcGIS object. File only.

_sharing_permissions

Return the level of access to the item (private, shared, org, public). A ‘shared’ item is an item that has its sharing level set to private but has been shared to groups. File only.

_webservice_url

Return the URL which hosted feature services can be accessed at. Only valid for feature-service like ArcGIS items.

_tags{}

A list attribute which returns the tags associated with an item. File only.

Publish an existing ArcGIS item as a feature service. ArcGIS item types which the connector supports for publish:

  • Shapefile
  • CSV
  • Microsoft Excel
  • Feature Collection
  • GeoJson
  • File Geodatabase
  • Scene Package
  • SQLite Geodatabase
  • Image Collection
  • Service Definition
  • Tile Package
  • Compact Tile Package
  • Vector Tile Package

Extra Usage Notes for Publish:

  • csv/excel files will be analyzed and have some publishing parameters autogenerated for them. The generated publishing parameters will be logged at the info level.
  • For CSV files, ArcGIS will attempt to classify the file as containing either coordinate data, address data, lookup data, or no location data. If the CSV file is classified as having location data but ArcGIS cannot determine the relevant fields containing this data, the ArcGISOnlineConnector will log a warning and publish the CSV file as though it does not contain any location data. The resulting feature service will be a table.

Publishing Options

Content Source

The Item ID of a stored ArcGIS file to publish. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only publishable items should be displayed in the browser. The user must be the owner of this item.

Note:

  • The item selected should not already have a feature service published from it. If a feature service already has been published from an item, the connector will reject and the id of the existing feature service will be contained in the _conflicting_id rejection attribute.

Feature Layer Title

The name to set for the newly created feature layer in ArcGIS. If this is left blank, the published item will use the existing title of the source item.

Note:

  • The title specified should be unique in the user’s organization.

Feature Layer Tags

The tags that the published feature layer should have in ArcGIS. If this is left blank, it published item will have the existing tags of the source item.

Coordinate System

Specify the target spatial reference of the created feature service. If not specified, this will default to WKID 102100.

Wait Behavior

Wait for Publish Processing

Whether to wait for the feature service created to publish

  • Yes: The connector will wait for the value specified in Processing Timeout and reject/pass the feature depending on the success of the publish operation.
  • No: The connector will make the publish request and pass/fail depending on the success of that request.

Processing Timeout (s)

The amount of time (seconds) to wait for the feature layer to publish. If not specified, this will default to 180 seconds.

Output Attributes

_id

Returns the ID of the feature service which was just created.

_private_url

Returns the link to the ArcGIS Online web content page for the newly created feature service

_service_type

Returns the type of service created with the publish operation

_webservice_url

Returns the URL which the newly created feature service can be accessed at. Only valid for feature-service like ArcGIS Online items.

Change the sharing level of a user’s ArcGIS Online content item or share it to additional groups.

Content Source

Path

The Item ID of an ArcGIS file to share. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content.

Sharing

Share Level

Specify the sharing level to set for the item.

  • Owner: The item will only be accessible to the user (and any groups it has been shared to).
  • Organization: The item will be accessible to the user and anyone else in their organization. If a user does not belong to an organization, behavior for this option is undefined.
  • Everyone (Public): The item will be visible to everyone. Sharing to everyone will automatically share to the organization as well, unless a user does not belong to an organization.

Share to Additional Groups

Select additional groups to share the item to. If there is a valid web connection, a list of all the groups the user belongs to will be displayed. Groups are listed with space separated IDs.

Notes:

  • The share action only shares to additional groups - if an item has already been shared to a group and it is not specified when the translation is run, the item will still remain shared to the group.
  • If one or more groups could not be shared to, the connector will reject the feature and log the groups which could not be shared to. If valid group_ids were specified, the item will still be shared to those groups.

Update the data stored for an existing ArcGIS item and optionally specify new title/tags for that item.

Content Source

Path

The Item ID of an ArcGIS file or folder to update. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only items which have data that can be updated by the connector will be displayed. If the user has permission to access items stored in other locations, the id of those items can be manually entered in this field.

Data Source

Upload

The type of data to update with.

  • File: The specified file will be uploaded as the new content.
  • Attribute: The contents of the specified attribute will be uploaded as the new content.

When working with large objects, File is a better choice than Attribute as the data will be streamed directly from disk and not require that the object be stored entirely in memory on a feature.

File to Upload

The file to be uploaded to ArcGIS. Valid if Uploading from File. The file type (file extension) must match that of the target update item, otherwise the feature will be rejected.

Attribute to Upload as File

The attribute to upload data from. Valid if Uploading from Attribute.

Metadata

Title

Specify a new title for the target item. If this is left blank, the title will not change.

Tags

Specify new tags for the target item. If this is left blank, the tags will not change. Tags should be provided as a comma separated list.

Output Attributes

_id

Returns the ID of the target item which was just created.

_private_url

Returns the link to the ArcGIS web content page for the updated item

Upload a file or attribute to a location in a user’s ArcGIS Content

Data Source

Upload

The type of data to be uploaded.

  • File: The specified file will be uploaded.
  • Attribute: The contents of the specified attribute will be uploaded.

When working with large objects, File is a better choice than Attribute as the data will be streamed directly from disk and not require that the object be stored entirely in memory on a feature.

File to Upload

The file to be uploaded to an ArcGIS account. Valid if Uploading from File.

Attribute to Upload as File

The attribute to upload data from. Valid if Uploading from Attribute.

Metadata

Title

The name to set for the uploaded item.

Item Type

The ArcGIS item type that the uploaded file should have.

Notes:

  • Where possible, the ArcGISOnlineConnector will attempt to use the extension in the provided filename/file path to determine the item type of the uploaded item. There are multiple valid ArcGIS item types for zip files, so users are manually required to select the item type for these items. If no valid item type can be determined or the file extension does not match a claimed type, the connector will reject the feature.

  • For a list of item types which ArcGIS Online supports for upload, see What can you add to ArcGIS Online?

Tags

Specify tags for the created item. Tags should be provided as a comma separated list.

Content Destination

User Content Folder

The ArcGIS folder ID to upload to.

The name of a user’s ArcGIS Folder can also be used, provided that it does not conflict with the ID of an existing folder. If no such folder exists, one with that name will be created.

If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only folders may be selected.

Note:

  • Filenames must be unique within ArcGIS content folders. If an item already exists in the folder with the same filename, the connector will reject and the id of the conflicting item will be output in the _conflicting_id rejection attribute.

Upload with File Name

The name of the file created from the data supplied in Attribute to Upload as File. The name must include a filename extension (for example, .csv, .jpg, .zip).

Output Attributes

_id

Returns the ID of the target item which was just created.

_private_url

Returns the link to the ArcGIS web content page for the updated item.

Editing Transformer Parameters

Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.

Using the Text Editor

The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.

Text Editor

Using the Arithmetic Editor

The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.

Arithmetic Editor

Conditional Values

Set values depending on one or more test conditions that either pass or fail.

Parameter Condition Definition Dialog

Content

Expressions and strings can include a number of functions, characters, parameters, and more.

When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.

String Functions

These functions manipulate and format strings.

Special Characters

 A set of control characters is available in the Text Editor.

Math Functions

Math functions are available in both editors.
Date/Time Functions Date and time functions are available in the Text Editor.

Math Operators

These operators are available in the Arithmetic Editor.

FME Feature Functions

These return primarily feature-specific values.

FME Parameters

FME and workspace-specific parameters may be used.
Creating and Modifying User Parameters Create your own editable parameters.

Dialog Options - Tables

Transformers with table-style parameters have additional tools for populating and manipulating values.

Row Reordering

Enabled once you have clicked on a row item. Choices include:

  • Add a row
  • Remove a row
  • Move current row up one
  • Move current row down one
  • Move current row to top
  • Move current row to bottom

Cut, Copy, and Paste

Enabled once you have clicked on a row item. Choices include:

  • Cut a row - delete and copy to clipboard
  • Copy a row to the clipboard
  • Paste a row from the clipboard

Cut, copy, and paste may be used within a transformer, or between transformers.

Filter

Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output.

Import

Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers.

Reset/Refresh

Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers.

Note: Not all tools are available in all transformers.

Reference

Processing Behavior

Feature-Based

Feature Holding

No
Dependencies Esri Access
Aliases AGOLConnector, ArcGISPortalConnector
History Released FME 2020.1

FME Community

The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.

Search for all results about the ArcGISOnlineConnector on the FME Community.

 

Examples may contain information licensed under the Open Government Licence – Vancouver and/or the Open Government Licence – Canada.