CKANConnector

Connects to a CKAN open data portal to manage datasets and resources.

Typical Uses

Browsing and downloading public datasets hosted on a CKAN portal
Creating or updating metadata and file-based Resources

How does it work?

The CKANConnector connects to a CKAN open data portal using the CKAN Action REST API to perform various tasks on datasets and their associated resources:

Action	Task
Create Dataset	Create a new CKAN dataset.
Create Resource	Create a new CKAN resource associated with an existing dataset by uploading content from a file or attribute, or providing a URL.
Delete Dataset	Delete a CKAN dataset and its resources.
Delete Resource	Delete a specific resource.
Download Resource	Download a resource to a file or attribute.
List Datasets	List the details of all datasets on a CKAN portal.
List Resources	List the details of all resources of a specific CKAN dataset.
Update Dataset	Update the details (metadata) of a specific CKAN dataset.
Update Resource	Update the content of a specific resource.

Configuration

Input Ports

Output Ports

Output

Features with added attributes, as specified in parameters and according to Action:

Action	Output
Create Dataset	Input feature with a JSON representation of the new dataset.
Create Resource	Input feature with a JSON representation of the new resource.
Delete Dataset	Input feature unchanged.
Delete Resource	Input feature unchanged.
Download Resource	Input feature with details about the downloaded item.
List Datasets	Multiple features, one for each dataset in the specified location, with details about the items and JSON representation.
List Resources	Multiple features, one for each resource in the specified dataset, with details about the items and JSON representation.
Update Dataset	Input feature with a JSON representation of the updated dataset.
Update Resource	Input feature with a JSON representation of the updated resource.

Parameters

Request

Action

Select an operation to perform. Choices include:

Create Dataset
Create Resource
Delete Dataset
Delete Resource
Download Resource
List Datasets
List Resources
Update Dataset
Update Resource

Action > Create Dataset

Create Dataset

Name	Specify the name of the new CKAN dataset. Names must be unique on the CKAN portal and be alphanumeric with no spaces.
Title	(Optional) Specify a title for the new dataset.
Private	Select a privacy setting: Yes: The dataset is only visible to the logged-in user. No: The dataset is visible to everyone.
Author	(Optional) Provide an author name.
Author Email	(Optional) Provide an author email address.
Maintainer	(Optional) Provide the name of the dataset maintainer.
Maintainer Email	(Optional) Provide the email address of the maintainer.
License	(Optional) Specify a dataset license, which may be selected from the options available on the CKAN portal.
Notes	(Optional) Provide any notes, comments, or description for the dataset.
Source	(Optional) Specify any applicable URL regarding the data source.
Version	(Optional) Specify any version information for the dataset.
Tags	(Optional) Specify any tags for the dataset, which may be selected from the options available on the CKAN portal. Tags that do not already exist will be created.
Groups	(Optional) Specify any groups to assign the dataset to, which may be selected from the options available on the CKAN portal.
Organization	Specify the organization that controls the dataset, which may be selected from the options available on the CKAN portal.

Output Attributes

JSON Response Attribute

Name the attribute to contain the JSON object representing the dataset.

Action > Create Resource

Data Source

Data Source	Select the source type: Attribute: Upload the contents of an attribute to the CKAN FileStore. File: Upload a file to the CKAN FileStore. URL: Set the URL property of the resource instead of uploading content. Note that folder-based formats and datasets with multiple files can be zipped and uploaded as one File.
File to Upload	When Data Source is File, specify the file to be uploaded.
Attribute to Upload as File	When Data Source is Attribute, specify the attribute containing the content for upload.
Upload with File Name	When Data Source is Attribute, specify the filename, including extension, to contain the uploaded content.
URL	When Data Source is URL, provide the URL of the resource.

Resource Metadata

Dataset	Specify the ID of the dataset to contain the new resource.
Name	(Optional) Provide a name for the resource. If not specified, the filename or URL basename is used.
Description	(Optional) Provide a description of the resource.
Format	(Optional) Specify the file format of the resource.
MIME Type	(Optional) Specify the MIME type of the uploaded file or resource URL. If not specified, the type is inferred from the filename or URL.

Output Attributes

JSON Response Attribute

Name the attribute to contain the JSON object representing the resource.

Action > Delete Dataset

Delete Dataset

Dataset

Specify the ID of the CKAN dataset to delete, along with its resources.

Purge

Select a deletion method:

Yes: Permanently delete the dataset.
No (Default): The dataset is marked as deleted and no longer appears in search results (as in the CKAN web interface).

Output Attributes

No attributes are added for this Action.

Action > Download Resource

Content Source

Dataset	Specify the ID of the dataset containing the resource to download.
Resource	Specify the ID of the resource to download.

Destination

Download As	Select a destination type for downloaded items: Attribute: The downloaded object will be stored in the specified attribute. File: The downloaded file will be stored in the specified folder.
Download to Attribute	When Download as is Attribute, specify the attribute to contain the contents of the downloaded item.
Download to Folder	When Download as is File, specify the path to the folder that will store the downloaded item.
If File Exists	When Download as is File, select a method for handling existing files: Reject: Leave the existing file intact and output the feature via the <Rejected> port. Overwrite: Overwrite the existing file.

Output Attributes

Attributes to Add

Select any available attributes to be included on the output features:

_download_path

The absolute path to the downloaded Resource file.

Action > List Datasets

Output Attributes

Attributes to Add

Select any available attributes to be included on the output features:

id	Dataset identifier
metadata_created	Date and time when the dataset was created
metadata_modified	Date and time when the dataset was last modified
name	Name of the dataset
title	Title of the dataset
author	Name of the dataset’s author
author_email	Email address of the dataset’s author
creator_user_id	Identifier of the creating user
groups	Groups to which the dataset belongs
license_id	Unique identifier for the dataset’s license
license_title	Title of the dataset’s license
maintainer	Name of the dataset’s maintainer
maintainer_email	Email address of the dataset’s maintainer
notes	Description of the dataset
organization	Dataset’s owning organization
owner_org	Identifier of the dataset’s owning organization
private	Whether a dataset is private or not
relationships_as_object	List of relationship dictionaries in which the dataset is the object
relationships_as_subject	List of relationship dictionaries in which the dataset is the subject
state	Current state of the dataset, ie. active or deleted
tags	Dataset’s tags
type	Type of the dataset
url	URL of the dataset’s source

JSON Response Attribute

Name the attribute to contain the JSON object representing the dataset.

Action > List Resources

List Resources

Dataset

Specify the ID of the dataset to list the resources of.

Output Attributes

Attributes to Add

Select any available attributes to be included on the output features:

created	Date and time when the resource was created
format	File format of the resource
id	Identifier for the resource
last_modified	Date and time when the resource was last modified
name	Name of the resource
package_id	Identifier of the package (dataset) that the resource belongs to
url	URL of the resource
cache_last_updated	Date and time when resource cache was last updated
cache_url	URL of the resource cache
datastore_active	Whether or not the resource can be read in CKAN datastore
description	Resource description
hash	Checksum for the resource
metadata_modified	Date and time when the resource metadata was modified
mimetype	MIME type of the resource
mimetype_inner	MIME type of the inner files if the resource is a container
position	Index used for the dataset display order
resource_type	If data, then: file or api. Otherwise: documentation
revision_id	Identifier for the last state of an updated resource
size	Size of the resource
state	Current state of the resource, ie. active or deleted
url_type	One of url, upload, or datastore

JSON Response Attribute

Name the attribute to contain the JSON object representing the resource.

Action > Update Dataset

Update Dataset

Current metadata values are preserved unless a new value is provided. To clear an existing value, set it to null.

Dataset ID	Specify the ID of the dataset to update.
Name	(Optional) Specify the name of the new CKAN dataset. Names must be unique on the CKAN portal and be alphanumeric with no spaces.
Title	(Optional) Specify a title for the new dataset.
Private	(Optional) Select a privacy setting: Yes: The dataset is only visible to the logged-in user. No: The dataset is visible to everyone.
Author	(Optional) Provide an author name.
Author Email	(Optional) Provide an author email address.
Maintainer	(Optional) Provide the name of the dataset maintainer.
Maintainer Email	(Optional) Provide the email address of the maintainer.
License	(Optional) Specify a dataset license, which may be selected from the options available on the CKAN portal.
Notes	(Optional) Provide any notes, comments, or description for the dataset.
Source	(Optional) Specify any applicable URL regarding the data source.
Tags	(Optional) Specify any tags for the dataset, which may be selected from the options available on the CKAN portal. Tags that do not already exist will be created.
Groups	(Optional) Specify any groups to assign the dataset to, which may be selected from the options available on the CKAN portal.
Organization	(Optional) Specify the organization that controls the dataset, which may be selected from the options available on the CKAN portal.

Output Attributes

JSON Response Attribute

Name the attribute to contain the JSON object representing the dataset.

Action > Update Resource

Data Source

Data Source	Select the source type: Attribute: Upload the contents of an attribute to the CKAN FileStore. File: Upload a file to the CKAN FileStore. URL: Set the URL property of the resource instead of uploading content. Note that folder-based formats and datasets with multiple files can be zipped and uploaded as one File.
File to Upload	When Data Source is File, specify the file to be uploaded.
Attribute to Upload as File	When Data Source is Attribute, specify the attribute containing the content for upload.
Upload with File Name	When Data Source is Attribute, specify the filename, including extension, to contain the uploaded content.
URL	When Data Source is URL, provide the URL of the resource.

Resource Metadata

Current metadata values are preserved unless a new value is provided. To clear an existing value, set it to null.

Dataset	Specify the ID of the dataset containing the resource.
Resource	Specify the ID of the resource to update.
Name	(Optional) Provide a name for the resource.
Description	(Optional) Provide a description of the resource.
Format	(Optional) Specify the file format of the resource.
MIME Type	(Optional) Specify the MIME type of the uploaded file or resource URL.

Output Attributes

JSON Response Attribute

Name the attribute to contain the JSON object representing the resource.

Editing Transformer Parameters

Transformer parameters can be set by directly entering values, using expressions, or referencing other elements in the workspace such as attribute values or user parameters. Various editors and context menus are available to assist. To see what is available, click beside the applicable parameter.

How to Set Parameter Values

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters.

Using the Text Editor

The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.

Text Editor

Using the Arithmetic Editor

The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.

Arithmetic Editor

Conditional Values

Set values depending on one or more test conditions that either pass or fail.

Parameter Condition Definition Dialog

Content

Expressions and strings can include a number of functions, characters, parameters, and more.

When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.

Content Types

String Functions	These functions manipulate and format strings.
Special Characters	A set of control characters is available in the Text Editor.
Math Functions	Math functions are available in both editors.
Date/Time Functions	Date and time functions are available in the Text Editor.
Math Operators	These operators are available in the Arithmetic Editor.
FME Feature Functions	These return primarily feature-specific values.
FME Parameters	FME and workspace-specific parameters may be used.
Creating and Modifying User Parameters	Create your own editable parameters.

Dialog Options - Tables

Table Tools

Transformers with table-style parameters have additional tools for populating and manipulating values.

Row Reordering

Enabled once you have clicked on a row item. Choices include:

Add a row
Remove a row
Move current row up one
Move current row down one
Move current row to top
Move current row to bottom

Cut, Copy, and Paste

Enabled once you have clicked on a row item. Choices include:

Cut a row - delete and copy to clipboard
Copy a row to the clipboard
Paste a row from the clipboard

Cut, copy, and paste may be used within a transformer, or between transformers.

Filter

Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output.

Import

Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers.

Reset/Refresh

Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers.

Note: Not all tools are available in all transformers.

For more information, see Transformer Parameter Menu Options.

Reference

Processing Behavior	Feature-Based
Feature Holding	No
Dependencies
Aliases
History

FME Community

The FME Community has a wealth of FME knowledge with over 20,000 active members worldwide. Get help with FME, share knowledge, and connect with users globally.

Search for all results about the CKANConnector on the FME Community.

Examples may contain information licensed under the Open Government Licence – Vancouver, Open Government Licence - British Columbia, and/or Open Government Licence – Canada.

CKANConnector

Typical Uses

How does it work?

Configuration

Input Ports

Output Ports

Parameters

Create Dataset

Output Attributes

Data Source

Resource Metadata

Output Attributes

Delete Dataset

Output Attributes

Delete Resource

Output Attributes

Content Source

Destination

Output Attributes

Output Attributes

List Resources

Output Attributes

Update Dataset

Output Attributes

Data Source

Resource Metadata

Output Attributes

Editing Transformer Parameters

Defining Values

Using the Text Editor

Using the Arithmetic Editor

Conditional Values

Content

Table Tools

Reference

FME Community