ArcGISOnlineConnector
Accesses the Esri ArcGIS Online content stored by a user as a file storage service to upload, update, download, delete, or list information about items from a user's Esri ArcGIS Online content. It can also be used to change the sharing settings of an item or publish items as hosted feature layers.
Typical Uses
- Manage datasets and supplementary files on ArcGIS Online by uploading, downloading, and deleting files and folders
- Transfer a file's contents into or out of an attribute in FME
- Read downloaded ArcGIS Online data using the FeatureReader, or upload data written by the FeatureWriter to ArcGIS Online
- Retrieve a user's file and folder names, paths, links and other information from ArcGIS Online to use elsewhere in a workspace.
- Delete items or folders from a user's ArcGIS Online content
- Download files or folders from either a user's own ArcGIS Online content or the content of a group they have access to
- List information about items and folders in either a user's own ArcGIS Online content or the content of a group they have access to
- Publish feature layers from valid items in a user's ArcGIS Online content
- Share items in a user's ArcGIS Online content to additional groups or change an item's sharing level
- Update the data files stored for existing items in a user's ArcGIS Online content
- Upload a file to ArcGIS Online and add it to a user's content
How does it work?
The ArcGISOnlineConnector uses your ArcGIS Online account credentials (either via a previously defined FME web connection, or by setting up a new FME web connection right from the transformer) to access the file storage service.
Depending on your choice of actions, it will upload or download files, folders, and attributes; list information from the service; share items which already exist in the user’s ArcGIS Online content; publish items as feature layers; or delete items from the service. On Uploads, link attributes are added to the output features. On List actions, links and file/folder information are added as attributes.
Examples
In this example, the ArcGISOnlineConnector is used to upload an Esri File Geodatabase from the user’s local files and publish it as a hosted feature layer.
The user can select the file geodatabase on their local system using the file picker in the ‘file to upload’ parameter. After creating a valid web connection to an ArcGIS Online account (which can be done in the ArcGIS Online Connection parameter), the user can specify the ArcGIS Online folder to upload the file to, as well as the title and tags to upload the item with.
After the file geodatabase has been successfully uploaded, you can use another ArcGISOnlineConnector to publish the newly created item. To keep the title and tags of the item you just created, you can leave those fields blank. You can also choose to wait for publish processing. This will wait for the specified amount of time while your new feature service processes.
You can also choose to use another ArcGISOnline connector to share the newly created feature service/uploaded item. By default, items will only be visible to yourself.
In this example, the ArcGISOnlineConnector is used to download an Esri Shapefile from the user’s ArcGIS Online Content. After creating a valid web connection to an ArcGIS Online account (which can be done in the ArcGIS Online Connection parameter), the user can browse their ArcGIS Online content and select their shapefile. They can also select a destination for the download.
A FeatureReader is added to read the newly downloaded dataset. The data from the downloaded shapefile is now available to use elsewhere in the workspace.
Usage Notes
- This transformer cannot be used to directly move or copy files between different ArcGIS Online user content folders or locations. However, multiple ArcGISOnlineConnectors can be used to accomplish these tasks.
- Uploading data to ArcGIS Online is performed via synchronous multipart upload. Each file chunk is 20MB. The connector will upload these chunks consecutively and if a chunk upload fails, the connector will retry each upload up to five times.
- Early termination of translations when performing publish/update/upload actions may result in malformed ArcGIS Online item entries. (When publishing, a feature service may be created but any title/tags specified for it may not be honored). (When updating/uploading, it is possible for the metadata to be uploaded before the file content has been uploaded).
- The FeatureReader can access ArcGIS Online directly (without using the ArcGISOnlineConnector), however, a local copy of the dataset will not be created.
Configuration
Input Ports
This transformer accepts any feature.
Output Ports
The output of this transformer will vary depending on the ArcGIS Online Action performed.
- A Delete action outputs no additional attributes.
- A Download action can save to a file, a folder, or an attribute. If it saves to a file or folder, it will output the path it saved to an attribute.
- A List action will output a new feature for each file/folder found in the path specified. Each of these new features will have attributes listing various pieces of information about the object.
- A Publish action will output information about the feature service it created, including its id, feature service type and URLs to the feature service itself as well as its ArcGIS Online content page.
- A Share action outputs no additional attributes.
- An Upload action will output the id of the uploaded item and a link to its ArcGIS Online content page.
- An Update action will output the id of the updated item and a link to its ArcGIS Online content page.
The incoming feature is output through this port.
Features that cause the operation to fail are output through this port. An fme_rejection_code attribute describing the category of the error will be added, along with a more descriptive fme_rejection_message attribute which contains more specific details as to the reason for the failure.
Note: If a feature comes in to the ArcGISOnlineConnector already having a value for fme_rejection_code, this value will be removed.
Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.
Parameters
ArcGIS Online Connection | Performing operations against an ArcGIS Online account requires the use of OAuth2 authentication using a Web Connection. To create an ArcGIS Online connection, click the 'ArcGIS Online Connection' drop-down box and select 'Add Web Connection...'. The connection can then be managed via Tools -> FME Options... -> Web Connections. |
Action |
The type of operation to perform. Choices include:
|
The remaining parameters available depend on the value of the Request > Action parameter. Parameters for each Action are detailed below.
Delete an existing item/folder from a user’s own ArcGIS Online content.
ArcGIS Online Content Source
Path |
The Item ID of a file or folder on ArcGIS Online to delete. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. The user must be the owner of this item. Notes:
|
Download a file/folder to an attribute/file/folder from a user’s ArcGIS Online content or a group they have access to.
ArcGIS Online Content Source
Content Source |
Select the source type of the content to be downloaded
|
Group ID |
The ArcGIS Online group which the content is linked to. Note:
|
Path |
The Item ID of a file or folder on ArcGIS Online to download. If valid credentials have been provided as a web connection, you may browse to a location. Depending on the value of the Content Source parameter, this will list either the contents of the specified group or the user’s own ArcGIS Online content. Only items which can be downloaded will be displayed. If the user has permission to access items stored in other locations, the id of those items can be manually entered in this field. |
Destination
Download as |
Select whether to store the downloaded data in a File, Folder or Attribute.
|
Download to Attribute |
Specify the attribute that will store the contents of the downloaded file. Valid for Download as Attribute only. Note: FME will attempt to convert the attribute's contents to a UTF-8 string. If this fails, the attribute's contents will be created as raw binary. To ensure that the original bytes are always preserved and never converted to UTF-8, use Download as File instead. |
Download to Folder | Specify the path to the folder that will store the downloaded file. Valid for Download as File or Folder. |
If File Exists | Whether to overwrite existing files in the target destination. Valid for Download as File or Folder. |
Include Subfolders | Whether to include subfolders when downloading a folder. Due to the way that ArcGIS Online content is structured, only the root folder contains subfolders. Valid for Download as Folder |
Output Attributes
_download_path | Return the path the file/folder was downloaded to. If a folder was downloaded, the path will have a trailing slash. |
List the item details for the contents of a group, a user’s folder, a user’s root folder, or all of a user’s content.
ArcGIS Online Content Source
Content Source |
Select the source type of the content to be listed
|
Group ID |
The ArcGIS Online group which the content is linked to. |
User Content Folder |
The folder ID on ArcGIS Online to list the contents of, or ‘/’ for root. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only folders may be selected. Valid when Content Source is set to User |
Include Subfolders |
Whether to include subfolder content when listing. Due to the way that ArcGIS Online content is structured, only the root folder contains subfolders. Valid when Content Source is set to User |
Output Attributes
_id |
Return the ID of an object on ArcGIS Online. |
_name |
Return the title of an object on ArcGIS Online. |
_file_or_folder |
Return the type (file or folder) of an object on ArcGIS Online. |
_private_url |
Return the link to an object on ArcGIS Online. If the object is an item, this will direct the user to the ArcGIS Online web content page for their item. If the object is a folder, this will direct the user to the ArcGIS Online web content listing of items for their folder. |
_created |
Return the creation time (UNIX timestamp) of an object on ArcGIS Online. |
_file_size |
Return the size of a file object (in bytes) on ArcGIS Online. File only. |
_file_type |
Return the file extension of a file object on ArcGIS Online. (for example, ‘zip’ or ‘csv’) File only, and only if the file is downloadable. |
_is_downloadable |
Returns a flag (Yes/No) depending on whether or not the ArcGIS Online connector can be used to download this object from AGOL. File only. |
_item_type |
Return the ArcGIS Online type of the file stored in ArcGIS Online. File only. |
_json_response |
Return the JSON response provided directly from the API for this item. |
_last_modified |
Return the last modified time (UNIX timestamp) of an object on ArcGIS Online. File only. |
_sharing_permissions |
Return the level of access to the item (private, shared, org, public). A ‘shared’ item is an item that has its sharing level set to private but has been shared to groups. File only. |
_webservice_url |
Return the URL which hosted feature services can be accessed at. Only valid for feature-service like ArcGIS Online items. |
_tags{} |
A list attribute which returns the tags associated with an item. File only. |
Publish an existing item in ArcGIS Online as a feature service. ArcGIS Online types which the connector supports for publish:
- Shapefile
- CSV
- Microsoft Excel
- Feature Collection
- GeoJson
- File Geodatabase
- Scene Package
- SQLite Geodatabase
- Image Collection
- Service Definition
- Tile Package
- Compact Tile Package
- Vector Tile Package
Extra Usage Notes for Publish:
- csv/excel files will be analyzed and have some publishing parameters autogenerated for them. They can be viewed as debug messages in the translation log
Publishing Options
ArcGIS Online Content Source |
The Item ID of a file or folder on ArcGIS Online to publish. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only publishable items should be displayed in the browser. The user must be the owner of this item. Note:
|
Feature Layer Title |
The name to set for the newly created feature layer in ArcGIS Online. If this is left blank, the published item will use the existing title of the source item. Note:
|
Feature Layer Tags |
The tags that the published feature layer should have in ArcGIS Online. If this is left blank, it published item will have the existing tags of the source item. |
Coordinate System |
Specify the target spatial reference of the created feature service. If not specified, this will default to WKID 102100. |
Wait Behavior
Wait for Publish Processing |
Whether to wait for the feature service created to publish
|
Processing Timeout (s) |
The amount of time (seconds) to wait for the feature layer to publish. If not specified, this will default to 180 seconds. |
Output Attributes
_id |
Returns the ID of the feature service which was just created. |
_private_url |
Returns the link to the ArcGIS Online web content page for the newly created feature service |
_service_type |
Returns the type of service created with the publish operation |
_webservice_url |
Returns the URL which the newly created feature service can be accessed at. Only valid for feature-service like ArcGIS Online items. |
Change the sharing level of a user’s ArcGIS Online content item or share it to additional groups.
ArcGIS Online Content Source
Path |
The Item ID of a file on ArcGIS Online to share. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. |
Sharing
Share Level |
Specify the sharing level to set for the item.
|
Share to Additional Groups |
Select additional groups to share the item to. If there is a valid web connection, a list of all the groups the user belongs to will be displayed. Groups are listed with space separated IDs. Notes:
|
Update the data stored for an existing item in ArcGIS Online and optionally specify new title/tags for that item.
ArcGIS Online Content Source
Path |
The Item ID of a file or folder on ArcGIS Online to update. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only items which have data that can be updated by the connector will be displayed. If the user has permission to access items stored in other locations, the id of those items can be manually entered in this field. |
Data Source
Upload |
The type of data to update with.
When working with large objects, File is a better choice than Attribute as the data will be streamed directly from disk and not require that the object be stored entirely in memory on a feature. |
File to Upload |
The file to be uploaded to ArcGIS Online. Valid if Uploading from File. The file type (file extension) must match that of the target update item, otherwise the feature will be rejected. |
Attribute to Upload as File |
The attribute to upload data from. Valid if Uploading from Attribute. |
Metadata
Title |
Specify a new title for the target item. If this is left blank, the title will not change. |
Tags |
Specify new tags for the target item. If this is left blank, the tags will not change. Tags should be provided as a comma separated list. |
Output Attributes
_id |
Returns the ID of the target item which was just created. |
_private_url |
Returns the link to the ArcGIS Online web content page for the updated item |
Upload a file or attribute to a location in a user’s ArcGIS Online Content
Data Source
Upload |
The type of data to be uploaded.
When working with large objects, File is a better choice than Attribute as the data will be streamed directly from disk and not require that the object be stored entirely in memory on a feature. |
File to Upload |
The file to be uploaded to ArcGIS Online. Valid if Uploading from File. |
Attribute to Upload as File |
The attribute to upload data from. Valid if Uploading from Attribute. |
Metadata
Title |
The name to set for the item uploaded to ArcGIS Online |
Item Type |
The ArcGIS Online item type that the uploaded file should have. Notes:
|
Tags |
Specify tags for the created item. Tags should be provided as a comma separated list. |
ArcGIS Online Content Destination
User Content Folder |
The folder ID on ArcGIS Online to list the contents of, or ‘/’ for root. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only folders may be selected. Note:
|
Upload with File Name | The name of the file created from the data supplied in Attribute to Upload as File. The name must include a filename extension (for example, .csv, .jpg, .zip). |
Output Attributes
_id | Returns the ID of the target item which was just created. |
_private_url | Returns the link to the ArcGIS Online web content page for the updated item. |
Editing Transformer Parameters
Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.
Defining Values
There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.
Using the Text Editor
The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.
Using the Arithmetic Editor
The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.
Conditional Values
Set values depending on one or more test conditions that either pass or fail.
Parameter Condition Definition Dialog
Content
Expressions and strings can include a number of functions, characters, parameters, and more.
When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.
These functions manipulate and format strings. | |
Special Characters |
A set of control characters is available in the Text Editor. |
Math functions are available in both editors. | |
Date/Time Functions | Date and time functions are available in the Text Editor. |
These operators are available in the Arithmetic Editor. | |
These return primarily feature-specific values. | |
FME and workspace-specific parameters may be used. | |
Creating and Modifying User Parameters | Create your own editable parameters. |
Dialog Options - Tables
Transformers with table-style parameters have additional tools for populating and manipulating values.
Row Reordering
|
Enabled once you have clicked on a row item. Choices include:
|
Cut, Copy, and Paste
|
Enabled once you have clicked on a row item. Choices include:
Cut, copy, and paste may be used within a transformer, or between transformers. |
Filter
|
Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output. |
Import
|
Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers. |
Reset/Refresh
|
Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers. |
Note: Not all tools are available in all transformers.
Reference
Processing Behavior |
|
Feature Holding |
No |
Dependencies | Esri Access |
FME Licensing Level | FME Base Edition and above |
Aliases | AGOLConnector |
History | Released FME 2020.1 |
FME Community
The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.
Search for all results about the ArcGISOnlineConnector on the FME Community.
Examples may contain information licensed under the Open Government Licence – Vancouver