CKAN DataStore Reader/Writer (FME Desktop Package)
Integrating this Format with FME Desktop
|
Download from FME Hub |
|
|
Other Package Components |
CKANConnector transformer |
Overview
The CKAN DataStore Reader/Writer allows FME to access DataStore tables stored on a CKAN server instance via the CKAN Action API. CKAN supports creating DataStore tables for any tabular data (for example, CSV/spreadsheets).
A CKAN dataset can contain multiple resources, some of which may have DataStore enabled. A writer/reader instance corresponds to one CKAN dataset. Each DataStore enabled resource corresponds to a single table in CKAN and a feature type in FME. Fields (columns) in a DataStore table correspond to attributes in FME and the records (rows) correspond to FME features.
When using the reader/writer, feature type names can be written as either <resource_name> (<resource_id) or <resource_id>. The id of a CKAN resource can be found either by using the CKANConnector transformer or by looking at the final part of a resource URL in the CKAN portal. Since resource names are not unique within a dataset, the resource id must be used to identify resources.
Authentication
Using the CKAN Reader/Writer requires a CKAN Web Connection. CKAN Connections define the CKAN Server URL and an optional API Token for authenticated access. Authenticated access is required for all operations with the writer and some with the reader.
An API token can be generated by logging into a CKAN web portal and going to a user profile, then clicking on API Tokens.
Reader Overview
The CKAN DataStore reader supports reading from all CKAN resources with an active DataStore table.
Note: If a resource does not have a table in DataStore, its file contents can still be retrieved through the CKANConnector parameter Action > Download Resource.
The reader will either read the entire contents of the table or filter the results using a WHERE clause. Support of WHERE clause querying is dependent on the CKAN server instance configuration; SQL search support must be enabled on the instance.
Writer Overview
The CKAN DataStore Writer supports writing to existing resources with a DataStore table and it can also create new CKAN resources within a dataset. The writer can be used to insert new rows and delete existing rows from CKAN resources. For tables with explicit primary keys, the writer can be used to perform Update/Upsert operations.
To create a table with an explicit primary key, set the User Attribute Index field for the Primary Key attribute(s) to a value of PrimaryKey. This table will now be eligible for use with the Upsert and Update operations.
If referring to an existing CKAN resource, a feature type name is expected to contain a resource id in the format <resource_name> (<resource_id) or <resource_id>. If a resource was originally uploaded as a file and a DataStore table has been created from its contents through usage of a tool such as DataPusher, it must be converted to a DataStore resource.
If a feature type name does not contain a resource id corresponding to a resource in the dataset, a new resource will be created and named after the feature type name.
The CKAN DataStore Writer does not have full transaction support, although it will attempt to roll back table/resource creation in the event of a write error (for example, invalid types contained in features). The translation will fail if a feature type specifies an attribute as a primary key/deletion filter and an attribute value for an incoming feature contains a null/missing value.