CKANConnector

Categories

Integrations

Web

Workflows

Related Transformers

AutodeskA360Connector

AzureBlobStorageConnector

AzureFileStorageConnector

BoxConnector

CesiumIonConnector

ComprehendConnector

DropboxConnector

FMEServerResourceConnector

GoogleCloudStorageConnector

GoogleDriveConnector

HDFSConnector

OneDriveConnector

RekognitionConnector

S3Connector

SQSConnector

Connects to a CKAN open data portal using the CKAN Action REST API. Supports reading and writing CKAN Dataset and Resource metadata, downloading of file-based Resources, and uploading files to the CKAN FileStore. Does not support row-based operations against CKAN DataStore Resources.

Jump to Configuration

Typical Uses

  • Browsing and downloading public datasets hosted on a CKAN portal
  • Creating or updating metadata and file-based Resources

Configuration

Input Ports

This transformer accepts any feature.

Output Ports

The output of this transformer will vary depending on the Request Action performed. Dataset and Resource metadata are presented as feature attributes.

The incoming feature is output through this port.

Features that cause the operation to fail are output through this port. Rejected features will have attributes fme_rejection_code and fme_rejection_message detailing the cause of rejection. Error responses from the CKAN server will have a fme_rejection_code value of ERROR_RESPONSE, with fme_rejection_message being the error messages returned by the server.

Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.

Parameters

CKAN Connection

Specifies the CKAN Connection to use. CKAN Connections define the CKAN Server URL and an optional API Token for authenticated access.
Action

The type of operation to perform. Choices are:

  • Create Dataset: Create a Dataset using the given metadata.
  • Create Resource: Create a Resource using the given metadata and/or file upload.
  • Delete Dataset: Delete or purge a Dataset.
  • Delete Resource: Permanently delete a specified Resource.
  • Download Resource: Given a Resource, request its URL and download the response to an attribute or folder.
  • List Datasets: List the Datasets on the CKAN portal. Each Dataset is a feature, and selected metadata fields are exposable as attributes.
  • List Resources: List the Resources of a selected Dataset. Each Resource is a feature, and selected metadata fields are exposable as attributes.
  • Update Dataset: Update a Dataset using the given metadata.
  • Update Resource: Update a Resource using the given metadata and/or file upload.

The remaining parameters available depend on the value of the Request > Action parameter. Parameters for each Action are detailed below.

Create Dataset

Name

Name of the dataset. Must be unique on the CKAN portal, and be alphanumeric with no spaces.

Title

Title of the Dataset.

Private

  • Yes: The Dataset is only visible to the logged in user.
  • No: The Dataset is visible to everyone.

Author

Name of the author.

Author Email

Email address of the author.

Maintainer

Name of the maintainer.

Maintainer Email

Email address of the maintainer.

License

Dataset license. Select from one of the options available on the CKAN portal.

Notes

Notes, comments, or description for the Dataset.

Source

The source URL for the Dataset.

Version

Version information for the Dataset.

Tags

Tags for the Dataset. Select from the options available on the CKAN portal. The server may create tags if they don’t already exist.

Groups

Assign the Dataset to these groups. Select from the options available on the CKAN portal.

Organization

Organization that controls the Dataset. Select from the options available on the CKAN portal.

Output Attributes

JSON Response Attribute

The JSON object representing the Dataset is stored in this attribute. Optional.

Data Source

Data Source

  • File: Upload a file for this Resource to the CKAN FileStore
  • Attribute: Upload an attribute’s data for this Resource to the CKAN FileStore.
  • URL: Set the URL property of the Resource instead of uploading a file.

File to Upload

Path to the file to upload.

Attribute to Upload as File

Name of attribute to upload.

Upload with File Name

When uploading an attribute, set this as the filename for when the Resource is downloaded.

URL

URL to set on the Resource.

Resource Metadata

All of the following parameters are optional, aside from the Dataset.

Dataset ID of the parent Dataset for this Resource.
Name Name of the Resource. If not specified, the value is set to the filename or URL basename.
Description Description of the Resource.
Format File format of the Resource.
MIME Type The MIME type of the uploaded file or Resource URL. If not specified, the value is inferred from the filename or URL.

Output Attributes

JSON Response Attribute

The JSON object representing the Resource is stored in this attribute. Optional.

Delete Dataset

Dataset

The ID of the Dataset to delete.

Purge

If Yes, the Dataset is permanently deleted and its metadata is no longer retrievable.

If No, the Dataset is marked as deleted and no longer appears in search results. This is how Delete works in the CKAN web interface.

Default: No

Delete Resource

Dataset

The ID of the Dataset that contains the Resource to delete.

Resource

The ID of the Resource to delete.

Content Source

Dataset

The ID of the Dataset that contains the Resource to download.

Resource

The ID of the Resource to download.

Destination

Download As
  • File: Download the Resource’s URL as a file.
  • Attribute: Download the Resource’s URL to an attribute.
Download to Attribute Download the Resource’s URL to this attribute.
Download to Folder Download the Resource’s URL to this folder. The filename is determined at the time of download.
If File Exists
  • Overwrite: Overwrite any existing destination file.
  • Reject: Reject the feature if the destination file already exists.

Output Attributes

Attributes to Add The _download_path attribute, if selected, contains the absolute path to the downloaded Resource file.

Output Attributes

Attributes to Add Sets the specified metadata field as attributes on the output feature.

JSON Response Attribute

The JSON object representing the Dataset is stored in this attribute. Optional.

List Resources

Dataset ID of the Dataset to list Resources from.

Output Attributes

Attributes to Add Sets the specified metadata field as attributes on the output feature.

JSON Response Attribute

The JSON object representing the Resource is stored in this attribute. Optional.

Update Dataset

When a field is left empty, the field’s existing value is left unmodified. To clear a field, set it to null.

Dataset ID

The ID of the Dataset to update.

Name

Name of the dataset. Must be unique on the CKAN portal, and be alphanumeric with no spaces.

Title

Title of the Dataset.

Private

  • Yes: The Dataset is only visible to the logged in user.
  • No: The Dataset is visible to everyone.

Author

Name of the author.

Author Email

Email address of the author.

Maintainer

Name of the maintainer.

Maintainer Email

Email address of the maintainer.

License

Dataset license. Select from one of the options available on the CKAN portal.

Notes

Notes, comments, or description for the Dataset.

Source

The source URL for the Dataset.

Tags

Tags for the Dataset. Select from the options available on the CKAN portal. The server may create tags if they don’t already exist.

Groups

Assign the Dataset to these groups. Select from the options available on the CKAN portal.

Organization

Organization that controls the Dataset. Select from the options available on the CKAN portal.

Output Attributes

JSON Response Attribute

The JSON object representing the Dataset is stored in this attribute. Optional.

Data Source

Data Source

  • File: Upload a file for this Resource to the CKAN FileStore
  • Attribute: Upload an attribute’s data for this Resource to the CKAN FileStore.
  • URL: Set the URL property of the Resource instead of uploading a file.

File to Upload

Path to the file to upload.

Attribute to Upload as File

Name of attribute to upload.

Upload with File Name

When uploading an attribute, set this as the filename for when the Resource is downloaded.

URL

URL to set on the Resource.

Resource Metadata

When a field is left empty, the field’s existing value is left unmodified. To clear a field, set it to null.

Dataset ID of the parent Dataset for this Resource.
Resource ID of the Resource to update.
Name Name of the Resource. If not specified, the value is set to the filename or URL basename.
Description Description of the Resource.
Format File format of the Resource.
MIME Type The MIME type of the uploaded file or Resource URL. If not specified, the value is inferred from the filename or URL.

Output Attributes

JSON Response Attribute

The JSON object representing the Resource is stored in this attribute. Optional.

Editing Transformer Parameters

Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.

Using the Text Editor

The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.

Text Editor

Using the Arithmetic Editor

The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.

Arithmetic Editor

Conditional Values

Set values depending on one or more test conditions that either pass or fail.

Parameter Condition Definition Dialog

Content

Expressions and strings can include a number of functions, characters, parameters, and more.

When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.

String Functions

These functions manipulate and format strings.

Special Characters

 A set of control characters is available in the Text Editor.

Math Functions

Math functions are available in both editors.
Date/Time Functions Date and time functions are available in the Text Editor.

Math Operators

These operators are available in the Arithmetic Editor.

FME Feature Functions

These return primarily feature-specific values.

FME Parameters

FME and workspace-specific parameters may be used.
Creating and Modifying User Parameters Create your own editable parameters.

Dialog Options - Tables

Transformers with table-style parameters have additional tools for populating and manipulating values.

Row Reordering

Enabled once you have clicked on a row item. Choices include:

  • Add a row
  • Remove a row
  • Move current row up one
  • Move current row down one
  • Move current row to top
  • Move current row to bottom

Cut, Copy, and Paste

Enabled once you have clicked on a row item. Choices include:

  • Cut a row - delete and copy to clipboard
  • Copy a row to the clipboard
  • Paste a row from the clipboard

Cut, copy, and paste may be used within a transformer, or between transformers.

Filter

Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output.

Import

Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers.

Reset/Refresh

Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers.

Note: Not all tools are available in all transformers.

Reference

Processing Behavior

Feature-Based

Feature Holding

No
Dependencies
Aliases  
History  

FME Community

The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.

Search for all results about the CKANConnector on the FME Community.

 

Examples may contain information licensed under the Open Government Licence – Vancouver and/or the Open Government Licence – Canada.