FME Transformers: 2024.1
FME Transformers: 2024.1
ArcGISOnlineConnector
Accesses Esri ArcGIS Online or Portal for ArcGIS to upload, manage, update, download, delete, publish, share, or list information about items from an Esri ArcGIS account.
Typical Uses
- Manage datasets and supplementary files on ArcGIS Online or Portal for ArcGIS by uploading, downloading, publishing, sharing, and deleting files and folders
- Transfer a file's contents into or out of an attribute in FME
- Read downloaded ArcGIS data using the FeatureReader, or upload data written by the FeatureWriter to ArcGIS Online or Portal for ArcGIS
- Retrieve file and folder names, paths, links and other information from ArcGIS Online or Portal for ArcGIS for use elsewhere in a workspace
-
Retrieve or update an item’s metadata
How does it work?
The ArcGISOnlineConnector uses either your ArcGIS Online or Portal for ArcGIS account credentials (either via a previously defined FME web connection, or by setting up a new FME web connection right from the transformer) to access the file storage service.
Depending on your choice of actions, it will upload or download files, folders, and attributes; list information from the service; share items which already exist in the user’s online content; publish items as feature layers; or delete items from the service. On Uploads, link attributes are added to the output features. On List actions, links and file/folder information are added as attributes.
Usage Notes
- This transformer cannot be used to directly move or copy files between different ArcGIS user content folders or locations. However, multiple ArcGISOnlineConnectors can be used to accomplish these tasks.
- Uploading data to ArcGIS is performed via synchronous multipart upload. Each file chunk is 20MB. The connector will upload these chunks consecutively and if a chunk upload fails, the connector will retry each upload up to five times.
- Early termination of translations when performing publish/update/upload actions may result in malformed ArcGIS item entries. (When publishing, a feature service may be created but any title/tags specified for it may not be honored). (When updating/uploading, it is possible for the metadata to be uploaded before the file content has been uploaded).
- The FeatureReader can access online ArcGIS content directly (without using the ArcGISOnlineConnector), however, a local copy of the dataset will not be created.
Configuration
Input Ports
Input
This transformer accepts any feature.
Output Ports
Output
The output of this transformer will vary depending on the Action performed.
- A Delete action outputs no additional attributes.
- A Download action can save to a file, a folder, or an attribute. If it saves to a file or folder, it will output the path it saved to an attribute.
-
A Get Details action can retrieve details or metadata for an item. The output feature will have attributes containing metadata values for the item.
- A List action will output a new feature for each file/folder found in the path specified. Each of these new features will have attributes listing various pieces of information about the object.
- A Publish action will output information about the feature service it created, including its id, feature service type and URLs to the feature service itself as well as its ArcGIS content page.
-
A Set Details action can update the details or metadata file for an item. The output feature will have the id of the updated item and a link to its ArcGIS content page.
- A Share action outputs no additional attributes.
- An Upload action will output the id of the uploaded item and a link to its ArcGIS content page.
- An Update action will output the id of the updated item and a link to its ArcGIS content page.
Initiator
The incoming feature is output through this port.
<Rejected>
Features that cause the operation to fail are output through this port. An fme_rejection_code attribute describing the category of the error will be added, along with a more descriptive fme_rejection_message attribute which contains more specific details as to the reason for the failure.
Note If a feature comes in to the ArcGISOnlineConnector already having a value for fme_rejection_code, this value will be removed.
Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.
Parameters
Authentication
ArcGIS Connection |
Performing operations against an ArcGIS Online or Portal for ArcGIS account requires the use of OAuth2 authentication using a Web Connection. To create an ArcGIS connection, click the 'ArcGIS Connection' drop-down box and select 'Add Web Connection...'. The connection can then be managed via Tools -> FME Options... -> Web Connections. To create an ArcGIS Portal Connection, users must create their own web connection based on the Esri ArcGIS Portal Web Service template provided with FME. From FME Workbench, open FME Options > Web Connections and click the Manage Services button. Navigate to the Esri ArcGIS Portal template. The resulting web connection can be used with the transformer. |
Request
Action |
The type of operation to perform. Choices include:
|
The remaining parameters available depend on the value of the Request > Action parameter. Parameters for each Action are detailed below.
Parameters - Delete Action
Delete an existing item/folder from a user’s own ArcGIS Online content.
Content Source
Path |
The Item ID of a file or folder on ArcGIS Online or Portal for ArcGIS to delete. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. The user must be the owner of this item. Notes:
|
Parameters - Download Action
Download a file/folder to an attribute/file/folder from a user’s ArcGIS content or a group they have access to.
Content Source
Content Source |
Select the source type of the content to be downloaded
|
Group ID |
The ArcGIS group which the content is linked to. Note:
|
Path |
The ArcGIS Item ID of a file or folder to download. If valid credentials have been provided as a web connection, you may browse to a location. Depending on the value of the Content Source parameter, this will list either the contents of the specified group or the user’s own ArcGIS content. Only items which can be downloaded will be displayed. If the user has permission to access items stored in other locations, the id of those items can be manually entered in this field. |
Destination
Download as |
Select whether to store the downloaded data in a File, Folder or Attribute.
|
Download to Attribute |
Specify the attribute that will store the contents of the downloaded file. Valid for Download as Attribute only. Note FME will attempt to convert the attribute's contents to a UTF-8 string. If this fails, the attribute's contents will be created as raw binary. To ensure that the original bytes are always preserved and never converted to UTF-8, use Download as File instead.
|
Download to Folder |
Specify the path to the folder that will store the downloaded file. Valid for Download as File or Folder. |
If File Exists |
Whether to overwrite existing files in the target destination. Valid for Download as File or Folder. |
Include Subfolders |
Whether to include subfolders when downloading a folder. Due to the way that ArcGIS content is structured, only the root folder contains subfolders. Valid for Download as Folder . |
Output Attributes
_download_path |
Return the path the file/folder was downloaded to. If a folder was downloaded, the path will have a trailing slash. |
Parameters - Get Details Action
Get details for an item, including its metadata.xml file.
Content Source
Content Source |
Select the source type of the content to be listed
|
Group ID |
The ArcGIS group which the content is linked to. |
Path |
The Item ID of a file on ArcGIS Online or Portal for ArcGIS to retrieve metadata for. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. |
Output Attributes
_id |
Return the ID of a stored ArcGIS object. |
_name |
Return the title of a stored ArcGIS object. |
_item_type |
Return the ArcGIS type of a stored ArcGIS object. File only. |
_tags |
Return a comma separated list of the tabs associated with an object. |
_summary |
Return the summary description of a stored ArcGIS object. |
_metadata_xml |
Returns the metadata.xml file associated with the ArcGIS object. If one does not exist, this attribute will be set to NULL. |
_description |
Return the description of a stored ArcGIS object. |
_type_keywords |
Return a comma separated list of keywords which describe the type of the object. This includes keywords determined based on its ArcGIS item type as well as user defined keywords, if any exist. |
_thumbnail_url |
Return a link which can be used to access the thumbnail image for the item. |
_credits |
Returns attribution information for the ArcGIS object as well as its copyright status |
_access_use_constraints |
Return license information or restrictions for the object. |
_extent |
Return the bounding box for an item. Coordinates are in WGS84 and are provided in the form [[minX, minY], [maxX, maxY]] |
_filename |
Return the name of the file associated with the stored object, if one exists. |
Parameters - List Action
List the item details for the contents of a group, a user’s folder, a user’s root folder, or all of a user’s content.
Content Source
Content Source |
Select the source type of the content to be listed
|
Group ID |
The ArcGIS group which the content is linked to. |
User Content Folder |
The ArcGIS folder ID to list the contents of, or ‘/’ for root. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only folders may be selected. Valid when Content Source is set to User. |
Include Subfolders |
Whether to include subfolder content when listing. Due to the way that ArcGIS content is structured, only the root folder contains subfolders. Valid when Content Source is set to User. |
Output Attributes
_id |
Return the ID of a stored ArcGIS object. |
_name |
Return the title of a stored ArcGIS object. |
_file_or_folder |
Return the type (file or folder) of a stored ArcGIS object. |
_private_url |
Return the link to an object on ArcGIS. If the object is an item, this will direct the user to the ArcGIS web content page for their item. If the object is a folder, this will direct the user to the ArcGIS web content listing of items for their folder. |
_created |
Return the creation time (UNIX timestamp) of a stored ArcGIS object. |
_file_size |
Return the size of a stored ArcGIS object (in bytes). File only. |
_file_type |
Return the file extension of a stored ArcGIS object. (for example, ‘zip’ or ‘csv’) File only, and only if the file is downloadable. |
_is_downloadable |
Returns a flag (Yes/No) depending on whether or not the ArcGISOnlineConnector can be used to download this object. File only. |
_item_type |
Return the ArcGIS type of a stored ArcGIS object. File only. |
_json_response |
Return the JSON response provided directly from the API for this item. |
_last_modified |
Return the last modified time (UNIX timestamp) of a stored ArcGIS object. File only. |
_sharing_permissions |
Return the level of access to the item (private, shared, org, public). A ‘shared’ item is an item that has its sharing level set to private but has been shared to groups. File only. |
_webservice_url |
Return the URL which hosted feature services can be accessed at. Only valid for feature-service like ArcGIS items. |
_tags{} |
A list attribute which returns the tags associated with an item. File only. |
Parameters - Publish Action
Publish an existing ArcGIS item as a feature service. ArcGIS item types which the connector supports for publish:
- Shapefile
- CSV
- Microsoft Excel
- Feature Collection
- GeoJson
- File Geodatabase
- Scene Package
- SQLite Geodatabase
- Image Collection
- Service Definition
- Tile Package
- Compact Tile Package
- Vector Tile Package
Extra Usage Notes for Publish:
- csv/excel files will be analyzed and have some publishing parameters autogenerated for them. The generated publishing parameters will be logged at the info level.
- For CSV files, ArcGIS will attempt to classify the file as containing either coordinate data, address data, lookup data, or no location data. If the CSV file is classified as having location data but ArcGIS cannot determine the relevant fields containing this data, the ArcGISOnlineConnector will log a warning and publish the CSV file as though it does not contain any location data. The resulting feature service will be a table.
Publishing Options
Content Source |
The Item ID of a stored ArcGIS file to publish. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only publishable items should be displayed in the browser. The user must be the owner of this item. Note:
|
Feature Layer Title |
The name to set for the newly created feature layer in ArcGIS. If this is left blank, the published item will use the existing title of the source item. Note:
|
Feature Layer Tags |
The tags that the published feature layer should have in ArcGIS. If this is left blank, it published item will have the existing tags of the source item. |
Coordinate System |
Specify the target spatial reference of the created feature service. If not specified, this will default to WKID 102100. |
Wait Behavior
Wait for Publish Processing |
Whether to wait for the feature service created to publish
|
Processing Timeout (s) |
The amount of time (seconds) to wait for the feature layer to publish. If not specified, this will default to 180 seconds. |
Output Attributes
_id |
Returns the ID of the feature service which was just created. |
_private_url |
Returns the link to the ArcGIS Online web content page for the newly created feature service |
_service_type |
Returns the type of service created with the publish operation |
_webservice_url |
Returns the URL which the newly created feature service can be accessed at. Only valid for feature-service like ArcGIS Online items. |
Parameters - Set Details Action
Update the details for an item or overwrite its metadata.xml file.
Content Source
Path |
The Item ID of a file on ArcGIS Online or Portal for ArcGIS to update metadata for. In order to successfully update the metadata, the user must own the item being updated. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. |
Details
Operation |
Select the operation to perform.
|
XML Metadata Attribute |
The attribute that contains XML content to upload as a metadata file. This parameter is only enabled if Operation is set to Replace Metadata XML. |
Fields
If a field is left blank, it will not be updated on the item. If it is set to NULL, the field will be cleared.
Title |
Specify a new title for the target item. |
Tags |
Specify new tags for the target item. Tags should be provided as a comma separated list. |
Summary |
Specify a new summary for the target item. |
Description |
Specify a new description for the target item. |
Credits |
Specify new attribution information for the target item as well as its copyright status. |
Access and Use Constraints |
Specify new license information or use restrictions for the target item. |
Type Keywords |
Specify new type keywords for the target item. Type keywords should be provided as a comma separated list. Note that updating this value only updates user-defined type keywords. Keywords associated with the ArcGIS Item Type will still remain. |
Thumbnail URL |
Set a new thumbnail for the target item. This should be provided using a URL to the image. |
Extent |
Set the bounding box for an item. Coordinates should be in WGS84 and must be provided in the form minX, minY,maxX, maxY. |
Output Attributes
_id |
Returns the ID of the target item which was updated. |
_private_url |
Returns the link to the ArcGIS web content page for the updated item. |
Parameters - Share Action
Change the sharing level of a user’s ArcGIS Online content item or share it to additional groups.
Content Source
Path |
The Item ID of an ArcGIS file to share. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. |
Sharing
Share Level |
Specify the sharing level to set for the item.
|
Share to Additional Groups |
Select additional groups to share the item to. If there is a valid web connection, a list of all the groups the user belongs to will be displayed. Groups are listed with space separated IDs. Notes:
|
Parameters - Update Action
Update the data stored for an existing ArcGIS item and optionally specify new title/tags for that item.
Content Source
Path |
The Item ID of an ArcGIS file or folder to update. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only items which have data that can be updated by the connector will be displayed. If the user has permission to access items stored in other locations, the id of those items can be manually entered in this field. |
Data Source
Upload |
The type of data to update with.
When working with large objects, File is a better choice than Attribute as the data will be streamed directly from disk and not require that the object be stored entirely in memory on a feature. |
File to Upload |
The file to be uploaded to ArcGIS. Valid if Uploading from File. The file type (file extension) must match that of the target update item, otherwise the feature will be rejected. |
Attribute to Upload as File |
The attribute to upload data from. Valid if Uploading from Attribute. |
Metadata
Title |
Specify a new title for the target item. If this is left blank, the title will not change. |
Tags |
Specify new tags for the target item. If this is left blank, the tags will not change. Tags should be provided as a comma separated list. |
Output Attributes
_id |
Returns the ID of the target item which was just created. |
_private_url |
Returns the link to the ArcGIS web content page for the updated item |
Parameters - Upload Action
Upload a file or attribute to a location in a user’s ArcGIS Content
Data Source
Upload |
The type of data to be uploaded.
When working with large objects, File is a better choice than Attribute as the data will be streamed directly from disk and not require that the object be stored entirely in memory on a feature. |
File to Upload |
The file to be uploaded to an ArcGIS account. Valid if Uploading from File. |
Attribute to Upload as File |
The attribute to upload data from. Valid if Uploading from Attribute. |
Metadata
Title |
The name to set for the uploaded item. |
Item Type |
The ArcGIS item type that the uploaded file should have. Notes:
|
Tags |
Specify tags for the created item. Tags should be provided as a comma separated list. |
Content Destination
User Content Folder |
The ArcGIS folder ID to upload to. The name of a user’s ArcGIS Folder can also be used, provided that it does not conflict with the ID of an existing folder. If no such folder exists, one with that name will be created. If valid credentials have been provided as a web connection, you may browse to a location in the user’s content. Only folders may be selected. Note:
|
Upload with File Name |
The name of the file created from the data supplied in Attribute to Upload as File. The name must include a filename extension (for example, .csv, .jpg, .zip). |
Output Attributes
_id |
Returns the ID of the target item which was just created. |
_private_url |
Returns the link to the ArcGIS web content page for the updated item. |
Editing Transformer Parameters
Transformer parameters can be set by directly entering values, using expressions, or referencing other elements in the workspace such as attribute values or user parameters. Various editors and context menus are available to assist. To see what is available, click beside the applicable parameter.
How to Set Parameter Values
Defining Values
There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters.
Using the Text Editor
The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.
Using the Arithmetic Editor
The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.
Conditional Values
Set values depending on one or more test conditions that either pass or fail.
Parameter Condition Definition Dialog
Content
Expressions and strings can include a number of functions, characters, parameters, and more.
When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.
Content Types
These functions manipulate and format strings. | |
Special Characters |
A set of control characters is available in the Text Editor. |
Math functions are available in both editors. | |
Date/Time Functions | Date and time functions are available in the Text Editor. |
These operators are available in the Arithmetic Editor. | |
These return primarily feature-specific values. | |
FME and workspace-specific parameters may be used. | |
Creating and Modifying User Parameters | Create your own editable parameters. |
Dialog Options - Tables
Table Tools
Transformers with table-style parameters have additional tools for populating and manipulating values.
Row Reordering
|
Enabled once you have clicked on a row item. Choices include:
|
Cut, Copy, and Paste
|
Enabled once you have clicked on a row item. Choices include:
Cut, copy, and paste may be used within a transformer, or between transformers. |
Filter
|
Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output. |
Import
|
Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers. |
Reset/Refresh
|
Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers. |
Note: Not all tools are available in all transformers.
For more information, see Transformer Parameter Menu Options.
Reference
Processing Behavior |
|
Feature Holding |
No |
Dependencies | Esri Access |
Aliases | AGOLConnector, ArcGISPortalConnector |
History | Released FME 2020.1 |
FME Community
The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.
Search for all results about the ArcGISOnlineConnector on the FME Community.
Examples may contain information licensed under the Open Government Licence – Vancouver, Open Government Licence - British Columbia, and/or Open Government Licence – Canada.