ArcGISOnlineConnector
Accesses Esri ArcGIS Online or Portal for ArcGIS to upload, manage, update, download, delete, publish, share, or list information about items from an Esri ArcGIS account.
Typical Uses
- Manage datasets and supplementary files on ArcGIS Online or Portal for ArcGIS by uploading, downloading, publishing, sharing, and deleting files and folders
- Transfer a file's contents into or out of an attribute in FME
- Read downloaded ArcGIS data using the FeatureReader, or upload data written by the FeatureWriter to ArcGIS Online or Portal for ArcGIS
- Retrieve file and folder names, paths, links and other information from ArcGIS Online or Portal for ArcGIS for use elsewhere in a workspace
How does it work?
The ArcGISOnlineConnector uses either your ArcGIS Online or Portal for ArcGIS account credentials (either via a previously defined FME web connection, or by setting up a new FME web connection right from the transformer) to access the file storage service.
Depending on your choice of actions, it will upload or download files, folders, and attributes; list information from the service; share items which already exist in the user’s online content; publish items as feature layers; or delete items from the service. On Uploads, link attributes are added to the output features. On List actions, links and file/folder information are added as attributes.
Examples
In this example, the ArcGISOnlineConnector is used to upload an Esri File Geodatabase from the user’s local files and publish it as a hosted feature layer.
The user can select the file geodatabase on their local system using the file picker in the ‘file to upload’ parameter. After creating a valid web connection to an ArcGIS Online account (which can be done in the ArcGIS Connection parameter), the user can specify the ArcGIS Online folder to upload the file to, as well as the title and tags to upload the item with.
After the file geodatabase has been successfully uploaded, you can use another ArcGISOnlineConnector to publish the newly created item. To keep the title and tags of the item you just created, you can leave those fields blank. You can also choose to wait for publish processing. This will wait for the specified amount of time while your new feature service processes.
You can also choose to use another ArcGISOnline connector to share the newly created feature service/uploaded item. By default, items will only be visible to yourself.
In this example, the ArcGISOnlineConnector is used to download an Esri Shapefile from the user’s ArcGIS Online Content. After creating a valid web connection to an ArcGIS Online account (which can be done in the ArcGIS Connection parameter), the user can browse their ArcGIS Online content and select their shapefile. They can also select a destination for the download.
A FeatureReader is added to read the newly downloaded dataset. The data from the downloaded shapefile is now available to use elsewhere in the workspace.
Usage Notes
- This transformer cannot be used to directly move or copy files between different ArcGIS user content folders or locations. However, multiple ArcGISOnlineConnectors can be used to accomplish these tasks.
- Uploading data to ArcGIS is performed via synchronous multipart upload. Each file chunk is 20MB. The connector will upload these chunks consecutively and if a chunk upload fails, the connector will retry each upload up to five times.
- Early termination of translations when performing publish/update/upload actions may result in malformed ArcGIS item entries. (When publishing, a feature service may be created but any title/tags specified for it may not be honored). (When updating/uploading, it is possible for the metadata to be uploaded before the file content has been uploaded).
- The FeatureReader can access online ArcGIS content directly (without using the ArcGISOnlineConnector), however, a local copy of the dataset will not be created.
Configuration
Input Ports
This transformer accepts any feature.
Output Ports
The output of this transformer will vary depending on the Action performed.
- A Delete action outputs no additional attributes.
- A Download action can save to a file, a folder, or an attribute. If it saves to a file or folder, it will output the path it saved to an attribute.
- A List action will output a new feature for each file/folder found in the path specified. Each of these new features will have attributes listing various pieces of information about the object.
- A Publish action will output information about the feature service it created, including its id, feature service type and URLs to the feature service itself as well as its ArcGIS content page.
- A Share action outputs no additional attributes.
- An Upload action will output the id of the uploaded item and a link to its ArcGIS content page.
- An Update action will output the id of the updated item and a link to its ArcGIS content page.
The incoming feature is output through this port.
Features that cause the operation to fail are output through this port. An fme_rejection_code attribute describing the category of the error will be added, along with a more descriptive fme_rejection_message attribute which contains more specific details as to the reason for the failure.
Note: If a feature comes in to the ArcGISOnlineConnector already having a value for fme_rejection_code, this value will be removed.
Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.
Parameters
ArcGIS Connection |
Performing operations against an ArcGIS Online or Portal for ArcGIS account requires the use of OAuth2 authentication using a Web Connection. To create an ArcGIS connection, click the 'ArcGIS Connection' drop-down box and select 'Add Web Connection...'. The connection can then be managed via Tools -> FME Options... -> Web Connections. To create an ArcGIS Portal Connection, users must create their own web connection based on the Esri ArcGIS Portal Web Service template provided with FME. From Workbench, open FME Options > Web Connections and click the Manage Services button. Navigate to the Esri ArcGIS Portal template. The resulting web connection can be used with the transformer. |
Action |
The type of operation to perform. Choices include:
|
The remaining parameters available depend on the value of the Request > Action parameter. Parameters for each Action are detailed below.
Delete an existing item/folder from a user’s own ArcGIS Online content.
Content Source
Path |
The Item ID of a file or folder on ArcGIS Online or Portal for ArcGIS to delete. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. The user must be the owner of this item. Notes:
|
Download a file/folder to an attribute/file/folder from a user’s ArcGIS content or a group they have access to.
Content Source
Content Source |
Select the source type of the content to be downloaded
|
Group ID |
The ArcGIS group which the content is linked to. Note:
|
Path |
The ArcGIS Item ID of a file or folder to download. If valid credentials have been provided as a web connection, you may browse to a location. Depending on the value of the Content Source parameter, this will list either the contents of the specified group or the user’s own ArcGIS content. Only items which can be downloaded will be displayed. If the user has permission to access items stored in other locations, the id of those items can be manually entered in this field. |
Destination
Download as |
Select whether to store the downloaded data in a File, Folder or Attribute.
|
Download to Attribute |
Specify the attribute that will store the contents of the downloaded file. Valid for Download as Attribute only. Note: FME will attempt to convert the attribute's contents to a UTF-8 string. If this fails, the attribute's contents will be created as raw binary. To ensure that the original bytes are always preserved and never converted to UTF-8, use Download as File instead. |
Download to Folder | Specify the path to the folder that will store the downloaded file. Valid for Download as File or Folder. |
If File Exists | Whether to overwrite existing files in the target destination. Valid for Download as File or Folder. |
Include Subfolders | Whether to include subfolders when downloading a folder. Due to the way that ArcGIS content is structured, only the root folder contains subfolders. Valid for Download as Folder |
Output Attributes
_download_path | Return the path the file/folder was downloaded to. If a folder was downloaded, the path will have a trailing slash. |
List the item details for the contents of a group, a user’s folder, a user’s root folder, or all of a user’s content.
Content Source
Content Source |
Select the source type of the content to be listed
|
Group ID |
The ArcGIS group which the content is linked to. |
User Content Folder |
The ArcGIS folder ID to list the contents of, or ‘/’ for root. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only folders may be selected. Valid when Content Source is set to User |
Include Subfolders |
Whether to include subfolder content when listing. Due to the way that ArcGIS content is structured, only the root folder contains subfolders. Valid when Content Source is set to User |
Output Attributes
_id |
Return the ID of a stored ArcGIS object. |
_name |
Return the title of a stored ArcGIS object. |
_file_or_folder |
Return the type (file or folder) of a stored ArcGIS object. |
_private_url |
Return the link to an object on ArcGIS. If the object is an item, this will direct the user to the ArcGIS web content page for their item. If the object is a folder, this will direct the user to the ArcGIS web content listing of items for their folder. |
_created |
Return the creation time (UNIX timestamp) of a stored ArcGIS object. |
_file_size |
Return the size of a stored ArcGIS object (in bytes). File only. |
_file_type |
Return the file extension of a stored ArcGIS object. (for example, ‘zip’ or ‘csv’) File only, and only if the file is downloadable. |
_is_downloadable |
Returns a flag (Yes/No) depending on whether or not the ArcGISOnlineConnector can be used to download this object. File only. |
_item_type |
Return the ArcGIS type of a stored ArcGIS object. File only. |
_json_response |
Return the JSON response provided directly from the API for this item. |
_last_modified |
Return the last modified time (UNIX timestamp) of a stored ArcGIS object. File only. |
_sharing_permissions |
Return the level of access to the item (private, shared, org, public). A ‘shared’ item is an item that has its sharing level set to private but has been shared to groups. File only. |
_webservice_url |
Return the URL which hosted feature services can be accessed at. Only valid for feature-service like ArcGIS items. |
_tags{} |
A list attribute which returns the tags associated with an item. File only. |
Publish an existing ArcGIS item as a feature service. ArcGIS item types which the connector supports for publish:
- Shapefile
- CSV
- Microsoft Excel
- Feature Collection
- GeoJson
- File Geodatabase
- Scene Package
- SQLite Geodatabase
- Image Collection
- Service Definition
- Tile Package
- Compact Tile Package
- Vector Tile Package
Extra Usage Notes for Publish:
- csv/excel files will be analyzed and have some publishing parameters autogenerated for them. The generated publishing parameters will be logged at the info level.
- For CSV files, ArcGIS will attempt to classify the file as containing either coordinate data, address data, lookup data, or no location data. If the CSV file is classified as having location data but ArcGIS cannot determine the relevant fields containing this data, the ArcGISOnlineConnector will log a warning and publish the CSV file as though it does not contain any location data. The resulting feature service will be a table.
Publishing Options
Content Source |
The Item ID of a stored ArcGIS file to publish. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only publishable items should be displayed in the browser. The user must be the owner of this item. Note:
|
Feature Layer Title |
The name to set for the newly created feature layer in ArcGIS. If this is left blank, the published item will use the existing title of the source item. Note:
|
Feature Layer Tags |
The tags that the published feature layer should have in ArcGIS. If this is left blank, it published item will have the existing tags of the source item. |
Coordinate System |
Specify the target spatial reference of the created feature service. If not specified, this will default to WKID 102100. |
Wait Behavior
Wait for Publish Processing |
Whether to wait for the feature service created to publish
|
Processing Timeout (s) |
The amount of time (seconds) to wait for the feature layer to publish. If not specified, this will default to 180 seconds. |
Output Attributes
_id |
Returns the ID of the feature service which was just created. |
_private_url |
Returns the link to the ArcGIS Online web content page for the newly created feature service |
_service_type |
Returns the type of service created with the publish operation |
_webservice_url |
Returns the URL which the newly created feature service can be accessed at. Only valid for feature-service like ArcGIS Online items. |
Change the sharing level of a user’s ArcGIS Online content item or share it to additional groups.
Content Source
Path |
The Item ID of an ArcGIS file to share. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. |
Sharing
Share Level |
Specify the sharing level to set for the item.
|
Share to Additional Groups |
Select additional groups to share the item to. If there is a valid web connection, a list of all the groups the user belongs to will be displayed. Groups are listed with space separated IDs. Notes:
|
Update the data stored for an existing ArcGIS item and optionally specify new title/tags for that item.
Content Source
Path |
The Item ID of an ArcGIS file or folder to update. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only items which have data that can be updated by the connector will be displayed. If the user has permission to access items stored in other locations, the id of those items can be manually entered in this field. |
Data Source
Upload |
The type of data to update with.
When working with large objects, File is a better choice than Attribute as the data will be streamed directly from disk and not require that the object be stored entirely in memory on a feature. |
File to Upload |
The file to be uploaded to ArcGIS. Valid if Uploading from File. The file type (file extension) must match that of the target update item, otherwise the feature will be rejected. |
Attribute to Upload as File |
The attribute to upload data from. Valid if Uploading from Attribute. |
Metadata
Title |
Specify a new title for the target item. If this is left blank, the title will not change. |
Tags |
Specify new tags for the target item. If this is left blank, the tags will not change. Tags should be provided as a comma separated list. |
Output Attributes
_id |
Returns the ID of the target item which was just created. |
_private_url |
Returns the link to the ArcGIS web content page for the updated item |
Upload a file or attribute to a location in a user’s ArcGIS Content
Data Source
Upload |
The type of data to be uploaded.
When working with large objects, File is a better choice than Attribute as the data will be streamed directly from disk and not require that the object be stored entirely in memory on a feature. |
File to Upload |
The file to be uploaded to an ArcGIS account. Valid if Uploading from File. |
Attribute to Upload as File |
The attribute to upload data from. Valid if Uploading from Attribute. |
Metadata
Title |
The name to set for the uploaded item. |
Item Type |
The ArcGIS item type that the uploaded file should have. Notes:
|
Tags |
Specify tags for the created item. Tags should be provided as a comma separated list. |
Content Destination
User Content Folder |
The ArcGIS folder ID to list the contents of, or ‘/’ for root. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only folders may be selected. Note:
|
Upload with File Name | The name of the file created from the data supplied in Attribute to Upload as File. The name must include a filename extension (for example, .csv, .jpg, .zip). |
Output Attributes
_id | Returns the ID of the target item which was just created. |
_private_url | Returns the link to the ArcGIS web content page for the updated item. |
Editing Transformer Parameters
Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.
Defining Values
There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.
Using the Text Editor
The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.
Using the Arithmetic Editor
The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.
Conditional Values
Set values depending on one or more test conditions that either pass or fail.
Parameter Condition Definition Dialog
Content
Expressions and strings can include a number of functions, characters, parameters, and more.
When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.
These functions manipulate and format strings. | |
Special Characters |
A set of control characters is available in the Text Editor. |
Math functions are available in both editors. | |
Date/Time Functions | Date and time functions are available in the Text Editor. |
These operators are available in the Arithmetic Editor. | |
These return primarily feature-specific values. | |
FME and workspace-specific parameters may be used. | |
Creating and Modifying User Parameters | Create your own editable parameters. |
Dialog Options - Tables
Transformers with table-style parameters have additional tools for populating and manipulating values.
Row Reordering
|
Enabled once you have clicked on a row item. Choices include:
|
Cut, Copy, and Paste
|
Enabled once you have clicked on a row item. Choices include:
Cut, copy, and paste may be used within a transformer, or between transformers. |
Filter
|
Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output. |
Import
|
Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers. |
Reset/Refresh
|
Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers. |
Note: Not all tools are available in all transformers.
Reference
Processing Behavior |
|
Feature Holding |
No |
Dependencies | Esri Access |
FME Licensing Level | FME Base Edition and above |
Aliases | AGOLConnector, ArcGISPortalConnector |
History | Released FME 2020.1 |
FME Community
The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.
Search for all results about the ArcGISOnlineConnector on the FME Community.
Examples may contain information licensed under the Open Government Licence – Vancouver and/or the Open Government Licence – Canada.