Licensing options for this format begin with FME Professional Edition.
Overview
FME provides read and write access to indexes on Elasticsearch clusters. FME treats each type in a Elasticsearch index as a feature type. Each document of a type is treated as a feature, and each field in a document is treated as an attribute.
Elasticsearch is an open source full-text search index. Elasticsearch indexes are JSON document stores that supports LonLat or GeoJSON geometry.
More information about Elasticsearch can be found at www.elastic.co.
Format Usage Notes
- There are two types of Elasticsearch geometry fields: geo_point and geo_shape. geo_point fields can only contain point geometries, while geo_shape fields can contain any geometry that is representable as GeoJSON.
- You can write features from most coordinate systems, but they will all be reprojected to LL-WGS84 when being converted to GeoJSON. The coordinate reference system for all GeoJSON coordinates is a geographic coordinate reference system, using the World Geodetic System 1984 (WGS 84) [WGS84] datum. [Reference: The GeoJSON Format]
- Writer: If a non-point geometry is written to a geo_point geometry field, then the geometry will be converted to its centroid point before writing.
- Writer: Each Elasticsearch document has a unique Document ID. This ID can be specified on a feature with an attribute selected in the Writer Feature Type Parameters. If a document with that ID already exists, then the translation will fail.
Reader Overview
The Elasticsearch reader supports reading multiple types from the same Elasticsearch index. Because of this, a separate reader must be created for each Elasticsearch index.
The types must be defined in the workspace before they can be read.
Multiple Geometry
The Elasticsearch reader supports reading multiple geometry fields from the same Elasticsearch type. If there is more than one geometry field in the type mapping, then geometry will be read as FME Multiple Geometry. Each geometry part will be named after the corresponding Elasticsearch geometry field.
Writer Overview
The Elasticsearch writer stores documents into a type associated with a Elasticsearch index. The Elasticsearch writer provides the following capabilities:
Type Creation
The Elasticsearch writer uses the information within the FME workspace to automatically create Elasticsearch types as required. A type will be created when the first input feature is processed. If no features are sent to a feature type, then the corresponding type will not be created.
Each Type is created with a Mapping (schema) based on the feature type’s User Attributes. The fields of each JSON document that is written to the Type will be parsed according to that Mapping. If the document contains any fields that do not appear in the Mapping, then those fields will be automatically added to it. This can occur if the Document Source of the feature type is a JSON Attribute.
Overwrite Index
If the Overwrite Index parameter for the writer is set to Yes, then the writer will drop and re-create the index before writing any features to it.
Indices will be overwritten when the first input feature is processed. If no features are sent to any of the writer’s feature types, then the corresponding index will not be overwritten.
Multiple Geometry
The Elasticsearch reader supports writing to multiple geometry fields in the same Elasticsearch type.
If there is more than one geometry field in the existing type mapping, then a feature's geometry must have the same name as destination Elasticsearch geometry field. Otherwise, no geometry will be written.
If a feature's FME Multiple Geometry has multiple parts, then each part can be written to a different Elasticsearch geometry field. Each part will be written to the Elasticsearch geometry field corresponding to its geometry part name, provided that the field exists.
Nested geometry fields can be created and/or written to by naming the geometry in the form:
<outer_name>.<inner_name>
For example, a geo_point geometry field called address.location would result in data similar to the following:
{
“address”: {
“location”: [ <lon>, <lat> ]
}
}