Metafile Directives
Used within the Universal Translator, Workbench, and FME Objects. Only
specific parts of the metafile get used by FME Objects.
Preambles
Workbench parses out the preamble sections for all readers and writers. These
lines end up in the workspace header (see: Workspace Structure), and this is
where Workbench gets the lines that end up in the navigation pane on the left
hand side of the Workbench window.
There are four preamble sections in total:
- SOURCE_PREAMBLE: Preamble for readers, goes into both Universal Translator
and Workbench translations.
- WORKBENCH_SOURCE_PREAMBLE: Preamble for readers, used in Workbench
translations, commented out in mapping files. Note that only the GUI line
gets commented out in mapping files, the rest (default macro and keyword) are
not commented out in a mapping file. This ensures that the default metafile
value will get used in both automated translations and in default workspaces.
- DESTINATION_PREAMBLE: Preamble for writers, goes into both Universal
Translator and Workbench translations.
- WORKBENCH_DESTINATION_PREAMBLE: Preamble for writers, used in Workbench
translations, commented out in mapping files. Note that only the GUI line
gets commented out in mapping files, the rest (default macro and keyword) are
not commented out in a mapping file. This ensures that the default metafile
value will get used in both automated translations and in default workspaces.
The reason we have two sets of preambles is because some obscure options may not
make sense in a quick Universal Translator drag-and-drop translation. These
options go into the WORKBENCH_ preamble, so users can customize the translation
further in Workbench. For a good example of this, see the SHAPE format metafile,
which has all four sections.
Preamble sections are often used to define macros and set source and destination
keywords. It is common practice to use a macro definition to define a value that
will be resolved at a later time. There are two ways to do this. MACRO defines a
value to set or override existing macros with the same name. DEFAULT_MACRO is
like MACRO but only supplies an initial value, and will not override existing
macros. Additionally, a key difference between these two macro definitions is
that DEFAULT_MACRO will automatically be changed based on the reader or writer
keyword, while MACRO will not.
DEFAULT_MACRO _ATTRKIND external_attributes
or
MACRO _ATTRKIND external_attributes
usage
$(_ATTRKIND)
One more note on this: You cannot assume that your preamble will be
all alone in a workspace. There may be two copies of it (say, you have
two AUTOCAD readers), so any items that need to be unique (say, a
factory name) need to be prefixed. You can use the WB_KEYWORD macro to
prepend to factory names or anything else of interest. See the AUTOCAD
source preamble metafile for an example of this.
SOURCE_READER
Syntax: SOURCE_READER <READER NAME> [<PARM NAME> <PARM_VALUE>] [[-]<Schema Keyword> <Schema Macro|Constant Value>]
This directive specifies the name of the reader used to drive the generation of
a mapping file/workspace. It also is where parameters specific to how the reader
is used are specified. Current, values include:
- SCHEMA_MODE LAYERS_ONLY
- This is only valid for DWG/DXF and causes a
mapping file to be generated which only uses the layer names from the input
file. Schema Scan directives are:
- IGNORE_ATTRIBUTES_WITH_PREFIX
- This directive identifies all of the attributes that are not to be used for the
purposes of generating a schema. There is no limit to the number of the
ignore statements which may be specified. For example, in AutoCAD there are
a large number of attributes that begin with extended_data. If we want to
ignore these then we simply define the following:
e.g. IGNORE_ATTRIBUTES_WITH_PREFIX extended_data
- KEEP_ATTRIBUTE
- This directive is used to identify attributes that are to be added to
the reader schema in all cases, when performing a FEATURE_SCAN. For
example, in AutoCAD the value is autocad_elevation. This indicates that no
matter if the autocad_elevation attribute is found in the feature scan,
the attribute is added to the schema.
- GEOMETRY_TYPE_FIELD
- This directive is used to identify the attribute that specifies the
different geometry types. For example, in AutoCAD the value is
autocad_entity. This is the value that controls the attribution specific
to a particular type of geometry.
- GEOMETRY_ATTRIBUTE_PREFIX
- This specifies the prefix for the attributes which are to be associated
with the geometry. For example, in autocad the attributes that begin with
autocad_ identify the geometry attributes.
- SCHEMA_MODE FEATURE_SCAN
- This value can be used for any reader and results in the source reader
reading all features. In this mode the features are scanned and a schema
is constructed. This is useful for formats that do not explicitly store
schema information. When this schema mode is specified then the reader's
readSchema method is not used.
- SCHEMA_MODE FILE_BASED_SCHEMA SCHEMA_FILE_NAME <basename>.sch
- This indicates the reader's schemas are canned/fixed/static, and that they
are defined in FME_HOME/schemas/<basename>.sch. As a result,
IFMEReader::readSchema() does not return any schema features. This is the
preferred method of implementing canned schemas, in opposition to achieving
the same result programmatically using IFMEReader::readSchema().
- SCAN_MAX_FEATURES <number>
- This causes the schema scanner to quit after receiving <number>
features, useful for large datasets when the entire schema is known after
a first few features.
Schema Keywords
Lastly, the SOURCE_READER line specifies keywords to be used during the reading of schema features.
The following conventions should be followed:
- The hyphen should be used.
- This means the keyword will now be in the mapping file, allowing us to share code used when
reading features instead of schemas
- In fact, the schema keyword won't even be present in the 'parameters' array passed through
IFMEReader::open(...)
- The schema keyword should be identical to the mapping file keyword. (i.e. READ_RAW_DATA)
- The macro name should be identical to the schema keyword. (i.e. READ_RAW_DATA)
- If a hyphen is used in front of the keyword, then the format short name prefix is not needed.
The infrastructure will prefix with a unique format keyword, so that -READ_RAW_DATA will result
in <unique format keyword>_READ_RAW_DATA, and can be fetched in code using _READ_RAW_DATA, which
will automatically be prefixed with the unique format keyword by a fetch call.
- Note that the keyword on the SOURCE_READER line will also work if it has a leading underscore
Here's an example - it has been adapted slightly from the current EDIGEO version because at the
time of writing, no format followed all these conventions
SOURCE_READER EDIGEO -READ_RAW_DATA $(READ_RAW_DATA) \
-READ_QUALITY_DATA NO \
-ADD_QUALITY_REFERENCE $(ADD_QUALITY_REFERENCE) \
-CNIG_NAME_COMPLETION $(CNIG_NAME_COMPLETION) \
-ERROR_OUTPUT $(ERROR_OUTPUT)
More detailed info
- When using the hyphen:
- in non-FME Objects situations, the values will not persist in the mapping file
object between readSchema() and read()
- this is because the mapping file that exists during readSchema() is a temporary
in-memory version,
and will get replaced by the physical mapping file produced by the core
- In FME Objects apps, the schema keyword will always be present in the mapping file,
regardless of whether the '-' was specified.
- In non-FME Objects, the keyword is only present when the '-' is specified. Results of
testing with FME Objects and schema keywords can be found here
- if you're curious, the code that processes the hyphen is in STFControlFormat::processSourceReader
- Schema keyword values containing unsafe (fme parser unsafe) characters:
- Use the following scheme to set the schema keywords. In this example _fme_encoded_ is a hint
to indicate that the following value is FMEParsableText encoded and '$encode' tells fme to FMEParsableText encode
macro's value before doing macro substitution.
Currently only "encode" and "decode" are supported for macro pre-processing. For a working
example see citygml.fmf and corresponding code for the CityGMLReader::readKeywordsHelper(),
where directives are fetched for schema reading.
<directiveName> _fme_encoded_$(<macroName>$encode)
E.g. SOURCE_READER XML -XML_XFMAP "_fme_encoded_$(XFMAP$encode)"
FORMAT_NAME <FORMAT NAME>
This directive simply defines the name of the format for which the file
describes the mapping file generation. This directive should always be
uppercase to ensure that FME and OEM licensing operate correctly.
SOURCE_KEYWORD <KEYWORD>
This is an optional directive. If not specified then the <format name> is used
as the keyword. This should rarely be used as the mapping file generation
always does the right thing. The only place where this is useful is perhaps for
SDE export in which case TABLE type is a bit non-obvious and confusing for the
novice reader. If this is not specified and the reader and writer are the same
the mapping file generation will generate a unique name of the form <format
name>_IN for this entry.
DESTINATION_KEYWORD <KEYWORD>
This is an optional directive. If not specified then the <format name> is
used as the keyword. This should rarely be used as the control file
generation always does the right thing. The only place where this is useful
is perhaps for SDE export in which case TABLE type is a bit non-obvious and
confusing for the novice reader. If this is not specified and the reader and
writer are the same the mapping file generation will generate a unique name
of the form <format name>_OUT for this entry.
SOURCE_DATASET <SOURCE DATASET>
This statement defines the value which is placed on the mapping file dataset
line. Typical values for this will be SOURCE_DATASET $[SourceDataset]
READER_META_ATTRIBUTES
This directive allows readers to selectively add fme_basename, fme_dataset,
and fme_feature_type attributes to all features they read.
For example, all raster readers specify
$(FMEGEN_SOURCE_KEYWORD)_READER_META_ATTRIBUTES fme_basename
so that the fme_basename attribute gets set on each feature.
DESTINATION_DATASET <DESTINATION DATASET>
This statement defines the value which is placed on the mapping file dataset
line. Typical values for this will be DESTINATION_DATASET $[DestDataset]
SOURCE_SETTINGS/END_SOURCE_SETTINGS
Enables the creation of metafile-based settings boxes for the reader. The
section must begin with SOURCE_SETTINGS, end with END_SOURCE_SETTINGS,
and must be placed outside the SOURCE_PREAMBLE/END_SOURCE_PREAMBLE
section.
The Settings Box Title will be placed in the title bar of the generated dialog.
The default title will be generated as: <Input|Output> Settings for <FormatLongName>
To override this default setting, add this line directly after the SOURCE_SETTINGS line:
GUI TITLE <Title>
This used to be mandatory, but now we just generate the title to make it
simpler.
Next, come the lines that will generate the GUI elements. If you don't
supply any of these lines then no settings box will be created, even if you
specify the GUI TITLE described above. For each setting, two lines must be
specified:
[-]DEFAULT_VALUE <keyword> <default value>
[-]GUI [OPTIONAL] <guiType> <keyword> <keyword description>
where:
- DEFAULT_VALUE line will result in a DEFAULT_MACRO line and a format
DIRECTIVE or KEYWORD line written to the mapping file/workspace file.
It would result in the following in a mapping file:
DEFAULT_MACRO x <default value>
<FMEGEN_<SOURCE|DESTINATION>_KEYWORD>_<keyword> "$(x)"
where x = <FMEGEN_<SOURCE|DESTINATION>_KEYWORD>_<IN|OUT>_<keyword>
While it would result in the following in a workspace:
DEFAULT_MACRO x <default value>
<WB_KEYWORD>_<keyword> "$(x)"
where x = <FMEGEN_<SOURCE|DESTINATION>_KEYWORD>_<IN|OUT>_<keyword>_<WB_KEYWORD>
For example, a line like this:
DEFAULT_VALUE UPPER_CASE_ATTR_NAMES Yes
would result in following two lines in a mapping file:
DEFAULT_MACRO SHAPE_IN_UPPER_CASE_ATTR_NAMES Yes
SHAPE_UPPER_CASE_ATTR_NAMES "$(SHAPE_IN_UPPER_CASE_ATTR_NAMES)"
And the same line would result in the following two lines in a workspace:
DEFAULT_MACRO SHAPE_IN_UPPER_CASE_ATTR_NAMES_SHAPE_1 Yes
SHAPE_1_UPPER_CASE_ATTR_NAMES "$(SHAPE_IN_UPPER_CASE_ATTR_NAMES_SHAPE_1)"
- When -DEFAULT_VALUE is used, the corresponding keyword will be accessible
during reading of schemas, but will not be written to the mapping file for
later use during reading of features. This is often used when certain
information is required for reading schemas (e.g., table names) that is not
required after the mapping file has been generated.
- <keyword> is the name of the keyword for which the setting will be used.
It is also useful to know that this is the name of the macro for the
keyword.
- <default value> is the desired default value. If no default is required,
then use "" (see textline.fmf). You must have a value or the parameters
from the settings box will not be replaced properly when workbench
generates the fmw.
- The macro made from this DEFAULT_VALUE line will not be created until the
metafile is parsed during mapping file or workspace file generation.
- The GUI line will be written to the mapping file exactly as it written
in the metafile. If -DEFAULT_VALUE is used then the corresponding GUI
line will be absent from the mapping file.
- If -GUI is used then GUI line will not be written to mapping file. This
is used if the keyword is to be set and used during schema generation, and
then cannot be changed when running the mapping file/workspace.
- [OPTIONAL] determines whether the field is optional. Omitting this token
makes the field mandatory. All mandatory fields must have some value,
otherwise the "Ok" button on the dialog will be disabled.
- <guiType> is one of the possible GUI lines directives. This includes
TEXT, PASSWORD, DIRNAME, FILENAME, CHOICE etc. If you use CHOICE, then
after <keyword> will be another field not described in the spec above in
order to simplify it: a '%' separated list of possible choices. For a
complete list of what directives to use go to FME GUI Directives. All the
directives from this page should apply except for ones that say
"proposed new types" or "Workbench specific types".
- <keyword description> is the label used for the setting.
- To have FEATURE_TYPES GUI directive work, you need to use two special
macro names. Macro DATASET is used to pass the value of dataset between
source/destination prompt and setting box. Macro TABLELIST is used by the
GUI framework to temporarily store the feature type list and then convert
to ID list for the reader. See example below. Similarly the macro
DESTINATION_TABLELIST is used for the special cases where the writer uses
the reader::readSchema() call to populate its schemas. See
WORKBENCH_CANNED_SCHEMA READ_AS_SOURCE for more information.
- Some custom settings boxes before allowed you to select the metafile used
in mapping file/workspace generation. This is enabled in metafile settings
boxes by the special macro METAFILE. The value returned for this macro will
be parsed out and returned correctly.
- Note: DEFAULT_VALUE <name> lines for macros that need to be used outside
the scope of the settings box section (SOURCE_SETTINGS/END_SOURCE_SETTINGS)
require a previous DEFAULT_MACRO <name> definition to work. The default
value specification should look like this, DEFAULT_VALUE <name> $(<name>).
- If your format has specialized connection parameters, then they should be
grouped in a NAMEDGROUP called FME_CONNECTION_GROUP. If the format is used
in a context where only connection information is needed (ex. SQLExecutor,
Joiner) then only parameters in that group will be shown.
You may have noticed that the second line of the two line pair has the same syntax
as other usage of GUI lines within the metafile.
Metafile comments can be inserted into this section by starting the line with '!'.
Settings Box Best Practices
We have some best practices regarding the look and feel of settings boxes. This helps
them look their best, and provides a consistent feel between all the settings boxes we
produce. They are as follows:
1. once you've begun using group boxes, you're committed to placing all widgets
within the current settings box within some group box
2. checkbox widgets must always be in groups, and at the top of the groups they're
in (this improves readability)
3. avoid ending group box descriptions with a colon
4. If your Reader
* only reads the specified feature types (has a required
Table List parameter), or
* has Feature Type parameters (e.g. SQL WHERE Clause) that
need to be applied to each feature type read via a merge
filter or Feature Types to Read
then your SOURCE_SETTINGS should include the following line:
DEFAULT_VALUE QUERY_FEATURE_TYPES_FOR_MERGE_FILTERS Yes
If your reader includes QUERY_FEATURE_TYPES_FOR_MERGE_FILTERS, and
your reader does not require a list DEF lines, because it
will read all features without any, then adding this line
to your metafile will skip the querying for 'dynamic' merged
data which is intended to read all features.
DEFAULT_VALUE ALL_MERGED_FILTERS_NEED_NO_DEFS Yes
This must be specified as a SOURCE_SETTINGS DEFAULT_VALUE
rather than a normal keyword because we need it to end up in
READER_GEN_DIRECTIVES.
Example
The following example is taken from JSON:
SOURCE_SETTINGS
!----------------------------------------------------------------------
! The title for the source settings dialog box.
!----------------------------------------------------------------------
GUI TITLE JSON Source Settings
!----------------------------------------------------------------------
! Delete the download file?
!----------------------------------------------------------------------
DEFAULT_VALUE DELETE_DOWNLOAD_FILE Yes
GUI CHOICE DELETE_DOWNLOAD_FILE Yes%No Delete downloaded file - [for URL datasets only]
!----------------------------------------------------------------------
! Get a proxy URL
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_URL ""
GUI OPTIONAL TEXT PROXY_URL Http Proxy URL:
!----------------------------------------------------------------------
! Get a proxy port number
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_PORT ""
GUI OPTIONAL TEXT PROXY_PORT Http Proxy Port:
!----------------------------------------------------------------------
! Get a proxy user name
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_USERNAME ""
GUI OPTIONAL TEXT PROXY_USERNAME Http Proxy Username:
!----------------------------------------------------------------------
! Get a proxy password
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_PASSWORD ""
GUI OPTIONAL PASSWORD PROXY_PASSWORD Http Proxy Password:
!----------------------------------------------------------------------
! Get a proxy authentication type
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_AUTH_METHOD Basic
GUI OPTIONAL CHOICE PROXY_AUTH_METHOD Basic%Digest Http Proxy Authentication Method:
END_SOURCE_SETTINGS
Example Feature Type Picker
SOURCE_SETTINGS
GUI TITLE Microsoft Access Input Settings
GUI NAMEDGROUP FME_CONNECTION_GROUP DATASET%PASSWORD Database Connection
GUI GROUP TABLELIST%WHERE_CLAUSE Constraints
# ===========================================================================
# The ADO provider in use. For Access, this is always MDB_ADO
DEFAULT_VALUE PROVIDER_TYPE MDB_ADO
# ===========================================================================
# This will show a widget to select dataset on the setting box. The
# preceding "-" will ensure that DEFAULT_MACRO line and KEYWORD line is not
# written out to the mapping file.
-DEFAULT_VALUE DATASET
-GUI FILENAME_MUSTEXIST DATASET Access_Database_Files(*.mdb)|*.mdb|All_files(*.*)|*.* Database Path:
# ===========================================================================
# Optional Password
DEFAULT_VALUE PASSWORD
GUI OPTIONAL PASSWORD PASSWORD Password:
# ===========================================================================
# This will show a feature type selector widget on the dialog. In order for it
# to work all the connection string parameters must be defined within the scope
# of this setting box.
-DEFAULT_VALUE TABLELIST
-GUI OPTIONAL FEATURE_TYPES TABLELIST $[DATASET],PROVIDER_TYPE,$[PROVIDER_TYPE],PASSWORD,$[PASSWORD] Table List:
# ===========================================================================
# Optional global WHERE clause
DEFAULT_VALUE WHERE_CLAUSE
GUI OPTIONAL TEXT WHERE_CLAUSE Where Clause:
END_SOURCE_SETTINGS
Example Search Envelope
# ===========================================================================
# To show search envelope settings this is how the GUI and DEFAULT_VALUE lines
# should like. Macro name MUST be like SEARCH_ENVELOPE_xxxx..., where "xxxx..." could
# could be used to make the macro names unique.
# ===========================================================================
-DEFAULT_VALUE USE_SEARCH_ENVELOPE NO
-GUI ACTIVEGROUP USE_SEARCH_ENVELOPE SEARCH_ENVELOPE_MINX%SEARCH_ENVELOPE_MINY%SEARCH_ENVELOPE_MAXX%SEARCH_ENVELOPE_MAXY%CLIP_TO_ENVELOPE Use Search Envelope
# this LOOKUP_GLOBAL has a drawback where the value of 0, if assigned to any parameters,
will be treated as if no value has been set to that parameter and a
# different value has to be used in UT. It is recommended to use "LOOKUP [parameter name] <Unused>, [value]"
for compatibility with UT
-GUI LOOKUP_GLOBAL <Unused>,0
-GUI LOOKUP CLIP_TO_ENVELOPE <Unused>,No
# ===========================================================================
DEFAULT_VALUE SEARCH_ENVELOPE_MINX 0
GUI FLOAT SEARCH_ENVELOPE_MINX Minimum X:
__
# ===========================================================================
DEFAULT_VALUE SEARCH_ENVELOPE_MINY 0
GUI FLOAT SEARCH_ENVELOPE_MINY Minimum Y:
__
# ===========================================================================
DEFAULT_VALUE SEARCH_ENVELOPE_MAXX 0
GUI FLOAT SEARCH_ENVELOPE_MAXX Maximum X:
__
# ===========================================================================
DEFAULT_VALUE SEARCH_ENVELOPE_MAXY 0
GUI FLOAT SEARCH_ENVELOPE_MAXY Maximum Y
__
# ===========================================================================
DEFAULT_VALUE CLIP_TO_ENVELOPE No
GUI CHECKBOX CLIP_TO_ENVELOPE Yes%No Clip to Search Envelope
Example Populating Dynamic Lists
See Hijacking the Feature Type Picker Dialog to Populate Dynamic Lists
DESTINATION_SETTINGS/END_DESTINATION_SETTINGS
This is the same as the SOURCE_SETTINGS/END_SOURCE_SETTINGS above except it
is for the writer. This means that the [-] option doesn't really make sense,
but it can be used. Some settings and includes are only intended for use
in the DESTINATION_SETTINGS and are often prefixed as such.
DESTINATION_DATASETTYPE_VALIDATION is a directive that relates to the
validation of the type of the destination dataset as compared to the
dataset type written in the formats.db.
For example, if the format has a dataset type of FILE in the formats.db
then enabling this directive means that the infrastructure before the
writer will ensure the dataset being passed to the writer is a valid file
prior to creating the writer. It does this by examining the dataset
value and determining whether it is a file or directory.
Note that only FILE, FILEDIR, FILEURLDIR or DIRONLY dataset types are
currently supported and that comparison only takes place if the
destination dataset is specified and exists. This check is not
relevant for any non-file based dataset formats, such as DATABASE,
URL, etc. as noted by the dataset type in the formats.db.
By default, a format does not perform this validation. As part of
format design, this directive can be added to help enforce the
validation of the dataset type.
Specifying the following include line in the DESTINATION_SETTINGS
section of a format metafile will cause validation to occur.
INCLUDE destinationDatasetTypeValidation.fmi
Note that some formats cannot leverage this directive if they are
currently, or have historically supported both FILE and DIR dataset
types, such as FFS, or if their dataset type is not file based.
DEF_LINE_TEMPLATE
This will determine the syntax for the DEF lines that mapping files and
workbench produce.
It is important that any parameters you define in the WORKBENCH_DEFLINE_PARMS
section appear here, or else they won't actually show up in the DEF lines. If
any parameters are defined on the DEF_LINE_TEMPLATE, Workbench will
automatically add everything defined in WORKBENCH_DEFLINE_PARMS.
If {FME_GEN_GEOMETRY} is provided on a def line, then DEF_LINE_BREAK must have
the value GEOM_CHANGE.
This provides a similar function in mapping files, with the exception that
WORKBENCH_DEFLINE_PARMS is irrelevant since it is only for Workbench; however
since your format should work in Workbench you must still use it.
DEF_LINE_TEMPLATE defines the format of the defline. It is of the form
DEF_LINE_TEMPLATE <groupName> [<def_attr> <def_attr_value>]*
where <def_attr> is the name of a definition attribute and <def_attr_value> is
the value of the definition line attribute. The value for group name may be
hardcoded or may be specified using one of two special keywords which are
available during the definition phase of the correlation table generation.
- {FME_GEN_GROUP_NAME} is the current name of the group as defined by the
input feature. The value of this macro is driven totally by the source side
of the format.
- {FME_GEN_GROUP_NUMBER} is the current group number. This macro is also
totally driven by the source side. This is useful when you want to ensure
that you assign unique numbers to different definition lines.
<def_attr_value> must be carefully handled in order to work in all possible
environments (Workbench, GENTRANS, UT Translate, UT Generate and Run). Based
on experimentation and the Geodatabase example present below, the rules are
as follows:
- Empty strings must be escape quote delimited: \"\"
- Strings with no spaces may be optionally quote delimited: "Object_ID" or Object_ID
- GENTRANS will treat escaped quotes around non-empty strings as significant.
- Strings with spaces must be quote and escape quote delimited: "\"Object ID\""
- Whether the def line template is being INCLUDEd may have an impact
on the quoting required.
DEF lines may identify the geometry columns via the fme_geometry_columns value.
This value has the following format:
[<FMEParsableText-encoded name>[,<type>];]+
This value should be used instead of format-specific directives
Here are a few examples:
MapInfo
DEF_LINE_TEMPLATE {FME_GEN_GROUP_NAME}
ARCGEN
DEF_LINE_TEMPLATE {FME_GEN_GROUP_NAME} \
ARCGEN_GEOMETRY {FME_GEN_GEOMETRY}
Shape
DEF_LINE_TEMPLATE {FME_GEN_GROUP_NAME}
SHAPE_GEOMETRY {FME_GEN_GEOMETRY}
Geodatabase
DEF_LINE_TEMPLATE {FME_GEN_GROUP_NAME} \
geodb_type {FME_GEN_GEOMETRY} \
GEODB_UPDATE_KEY_COLUMNS \"\" \
GEODB_OBJECT_ID_NAME "Object_ID" \
GEODB_OBJECT_ID_ALIAS "\"Object ID\"" \
GEODB_SHAPE_NAME "Shape" \
GEODB_SHAPE_ALIAS "Shape" \
GEODB_CONFIG_KEYWORD "DEFAULTS" \
GEODB_FEATURE_DATASET \"\" \
GEODB_GRID{1} \"\" \
GEODB_AVG_NUM_POINTS \"\" \
$(_GEODBOutDEF_Xorigin) \
$(_GEODBOutDEF_Yorigin) \
$(_GEODBOutDEF_XYscale) \
GEODB_HAS_Z_VALUES \"\" \
$(_GEODBOutDEF_Zorigin) \
$(_GEODBOutDEF_Zscale) \
GEODB_HAS_MEASURES \"\" \
$(_GEODBOutDEF_Morigin) \
$(_GEODBOutDEF_Mscale) \
GEODB_ANNO_REFERENCE_SCALE \"\"
DEF_LINE_BREAK
This affects how DEF lines are created within Workbench/Universal Translator (UT).
If the line contains:
- GEOM_CHANGE, then only single geometry per feature type will be allowed, and the 'Geometry'
combo box in Workbench will be enabled in the feature type properties dialog.
- ATTRIB_CHANGE, then each feature type can handle all geometry types, and 'Geometry' will be
disabled in the Workbench combo box.
- NO_DEF, no DEF lines will be written at all, and the geometry combo box in Workbench will be
disabled.
- NO_ATTRIBS, then no user-defined attributes will be written onto the DEF line. This also
means in Workbench, feature type properties dialog will not show the "User Attributes"
tab.
Note that a single line can contain more than one of these values in it (i.e. for BC MOEP the
line is: DEF_LINE_BREAK ATTRIB_CHANGE NO_ATTRIBS).
GEOM_MAP
This section defines how a format's types map into fme transfer types. For simplicity's sake, just think of transfer types as though they were fme types (i.e. fme_type). It is important that there are format types that map to each fme transfer type. The complete list of fme transfer types along with their permitted attributes is:
- fme_point
- fme_color
- fme_symbol (rarely used)
- fme_size (rarely used)
- fme_line
- fme_color
- fme_style (rarely used)
- fme_width (rarely used)
- fme_smooth_line (rarely used)
- fme_polygon
- fme_fill_style (rarely used)
- fme_foreground_color (rarely used)
- fme_background_color (rarely used)
- fme_color
- fme_style (rarely used)
- fme_width (rarely used)
- fme_text
- fme_rotation
- fme_oblique_angle (rarely used)
- fme_background_color (rarely used)
- fme_color
- fme_font (rarely used)
- fme_text_style (rarely used)
- fme_text_size
- fme_justification (rarely used)
- fme_text_string
- fme_spacing (rarely used)
- fme_text_width (rarely used)
- fme_ellipse
- fme_primary_axis
- fme_secondary_axis
- fme_rotation
- fme_fill_style (rarely used)
- fme_foreground_color (rarely used)
- fme_background_color (rarely used)
- fme_color
- fme_style (rarely used)
- fme_width (rarely used)
- fme_arc
- fme_primary_axis
- fme_secondary_axis
- fme_start_angle
- fme_sweep_angle
- fme_rotation
- fme_color
- fme_style (rarely used)
- fme_width (rarely used)
- fme_smooth_line (rarely used)
- fme_rectangle (rarely used)
- fme_fill_style (rarely used)
- fme_foreground_color (rarely used)
- fme_background_color (rarely used)
- fme_color
- fme_style (rarely used)
- fme_width (rarely used)
- fme_rounded_rectangle (rarely used)
- fme_rounding (rarely used)
- fme_fill_style (rarely used)
- fme_foreground_color (rarely used)
- fme_background_color (rarely used)
- fme_color
- fme_style (rarely used)
- fme_width (rarely used)
- fme_no_geom
- fme_collection (implied mapping: fme_no_geom)
- fme_point_cloud (implied mapping: fme_polygon)
- fme_raster (implied mapping: fme_polygon)
- fme_solid (implied mapping: fme_line)
- fme_surface (implied mapping: fme_line)
- fme_voxel_grid (implied mapping: fme_point_cloud)
It is absolutely valid that multiple fme transfer types map to the same format
type. For example, geodb_polygon maps to fme_polygon, fme_rounded_rectangle,
and fme_rectangle. It is also possible for a single fme transfer type to map
to multiple format types. Continuing the example with Geodatabase,
geodb_polyline, geodb_simple_edge, and geodb_complex_edge all map to fme_line.
In this situation however, the order of these mappings (i.e. which is first in
the file) is very important because when writing to the format, fme_line will
get mapped to the first match found in the metafile, which in the Geodatabase
case would be geodb_polyline.
Within Workbench, this section lists all the possible geometry types this
format supports. It is important that a mapping to all fme transfer types be
present, for if Workbench cannot map a generic type (say, fme_line) bad things
will happen.
Look at existing formats to see what this section looks like.
Some fme_type values have an "implied mapping". These types were added later,
and thus may not be present in all metafiles. If no explicit mapping is found,
the same mapping is used as for the implied type. For example, for mapinfo,
fme_raster will be mapped to mapinfo_region (the mapinfo mapping for
fme_polygon).
Note that fme_collection is special because it is used as an indicator whether
a writer has support for rich IFMEAggregate geometries or for heterogeneous
classic FME_GEOM_AGGREGATE/STF_GEOM_AGGREGATE geometries. if fme_collection
appears in a writer's geom_map, then heterogeneous IFMEAggregates and
homogeneous IFMEAggregates (that could alternatively be IFMEMultis) will be
send to the writer to handle. if fme_collection is not in a writer's geom_map,
then heterogeneous IFMEAggregates will be split into homogeneous IFMEMultis,
and homogeneous IFMEAggregates will be simplified into IFMEMultis.
Note also that a special value of fme_no_map can be used on the left
side of a GEOM_MAP line to indicate that there is no valid mapping for
the corresponding fme type on the right side.
ATTR_TYPE_MAP
This section defines how a format's attribute types map into fme
attribute types and vice versa.
Basically, the right-hand side lists all the possible attribute types that
FME understands. It is important that each of these types have a mapping
into the specific format attribute types used by your format. The
IFMEReader::readSchema() method should return the attribute types on the
left hand side on feature types that have user attributes.
Note that the ATTR_TYPE_MAP is read in one of two ways:
a. Source Format / Reader / Genericize - Left to Right, or
b. Dest Format / Writer / Specialize - Right to Left
Terminology
Case a. above: Reading left to right is called genericizing.
Case b. above: Reading right to left is called specializing.
The left side of the ATTR_TYPE_MAP is the format attribute types.
The right side of the ATTR_TYPE_MAP is the FME attribute types.
Arguments to types that exist in brackets are called descriptors.
In both cases above, entries are read top down to find a match.
Exact matches are expected for types without descriptors, and
otherwise either types with specific descriptors, I.E. char(20), or
types with placeholder descriptors, I.E. char(width) are mapped to
the best possible match.
A good starting point for the ATTR_TYPE_MAP would be:
ATTR_TYPE_MAP string fme_buffer \
char(width) fme_varchar(width) \
char(width) fme_char(width) \
datetime fme_datetime \
time fme_time \
date fme_date \
real64 fme_real64 \
real32 fme_real32 \
int32 fme_int32 \
uint32 fme_uint32 \
int64 fme_int64 \
uint64 fme_uint64 \
logical fme_boolean \
int16 fme_int16 \
uint16 fme_uint16 \
uint8 fme_uint8 \
number(width,decimal) fme_decimal(width,decimal) \
string fme_xml \
string fme_json
Note that the format types are arbitrary and are named by the format
creator. Format creators are encouraged to to name format types
based on native format names, as best as possible, and to be as
explicit as possible with types so that they can be addressed
individually if needed. Which usually means listing all supported
format types.
Note: fme_char used to be a first-class fme type, but has since been
replaced by fme_uint8, or just fme_char(width)
WARNING: Attempting to modify the attribute type map after the
original definition can lead to serious compatibility issues. Should you
need to modify the map, it is strongly recommended that a type name be
never removed from the map. Move the old type name lower in the mapping
order.
In particular, it is important to specify date type fields.
If formats explicitly identify date, time or datetime types,
then, when translating to database or other formats that require
specific formatting, date types can be detected and the date values
can be properly formatted.
However, the format creator is responsible for ensuring that date,
time, or datetime attribute values are formatted correctly according
to the FME date, time and datetime standards when reading as well.
In general, the format attribute types should not be prefixed by
some format identifier(e.g. use "date" instead of "sdf_date"),
unlike with the GEOM_MAP entries.
Specialization Only Mappings
Specific format attributes that contain numbers in parenthesis or
hyphens as suffixes (e.g. char(20), number(10,10), or text-)
are only used for specializing.
For example, the following mapping
number(31,15) fme_real64
or from the example above:
char(1) fme_char
will not have any effect if this is the source format and we are
genericizing the attribute types.
Unsupported FME Types
It is possible that a format does not support one or more of the FME
attribute types. In this case, it can be mapped to an appropriate
attribute type that the format does support.
For example, you mapping might look like Shape's/DBF's (dbftypes.fmi):
ATTR_TYPE_MAP char(width) fme_varchar(width) \
char(width) fme_char(width) \
$(DBF_TYPES) \
char(254) fme_buffer \
char(20) fme_datetime \
char(12) fme_time \
char(1) fme_char \
date fme_date \
double fme_real64 \
double fme_uint32 \
float fme_real32 \
number(20,0) fme_int64 \
number(20,0) fme_uint64 \
logical fme_boolean \
short fme_int16 \
short fme_uint8 \
long fme_int32 \
long fme_uint16 \
number(width,decimal) fme_decimal(width,decimal)
Notice that fme_buffer, fme_datetime, and fme_time all map to
char(n) of a specific width large enough to store that type of value.
Additionally, both signed and unsigned 64 bit floating point numbers
map to number(20,0).
Descriptor Matching
There are many valid mappings possible for each type. For example,
an FME attribute type of fme_decimal(width, decimal) could be
defined to map to any of the following format types.
Example valid mappings:
number(20,10) fme_decimal(width,precision)
number(width,decimal) fme_decimal(width,precision)
char(20) fme_decimal(width,precision)
char(width) fme_decimal(width,precision)
text- fme_decimal(width,precision)
However, in some case we want more complex mappings such as the
cases below. Most of these arise from cases with mismatched
numbers or values for descriptors beween the format and fme
attribute types.
a. Discarding Undesirable Type Descriptors
This is possible using hyphens on the format attribute and are
useful for specialization only.
If the right side has a (width) or (width, decimal) and you cannot/do not want
to use these descriptors then use a hyphen on the left side.
double- fme_decimal(width,decimal)
Essentially the hyphen means that the width, decimal will be thrown away.
Note that this also means that the mapping will only be used when
writing, where we map from the generic fme attribute type on the right
to the format specific attribute type on the left. The hyphen will
prevent the line above from being used as a valid mapping when reading.
b. Matching Unequal Numbers of Descriptors
It is also possible to create mappings where specializing will
map types with different numbers of descriptors.
If specializing and the FME attribute type has MORE descriptors
than the format attribute type, only the required number of
attributes are taken.
Map : char(width) fme_decimal(width,decimal)
Value: fme_decimal(31,15) -> char(31)
If specializing and the FME attribute type has LESS descriptors
than the format attribute type, the descriptors would be taken
literally from the FME type, which is the default behavior.
Map : string(width,xlsx_col_props) fme_varchar(width)
Value: fme_varchar(21) -> string(21)
None of these format types could be used for genericizing.
c. Example Descriptor Mismatches
Examples of actual literal values passing through the
ATTR_TYPE_MAP would behave as follows:
Map : char(255) fme_decimal(width,decimal)
Value: fme_decimal(31,15) -> char(255)
Map : char(width) fme_decimal(width,decimal)
Value: fme_decimal(31,15) -> char(31)
Map : string(width,xlsx_col_props) fme_varchar(width)
Value: fme_varchar(21) -> string(21)
Exact Attribute Matching
In addition to the regular specialize and genericize steps
there is an extra post mapping step involved when both the source
and destination formats are known, which is sometimes referred to
as end-to-end mapping. This extra step is useful when generating
non-default types to the same format or to another format that
supports attributes of the exact same type.
The exact matching will attempt to see if there is a better type
than the one initially found in the base mapping by checking to
see if the exact source format type can be found in the destination
attribute types.
Note that the exact attribute type name must exist
on the format side of the ATTR_TYPE_MAP of both the source and
destination formats and they must map to the same fme attribute
type as well. If so, the original type is preferred and replaces
the previous mapped type from genericization and specialization.
For example, let's say that an input attribute type of char(50)
initially genericized to fme_char(width) and then specialized to
string- by default in the destination. The exact attribute type
matching would might observe that the destination format also has
a char(width) type. If this is true, then the input attribute type
of char(50) will instead be mapped to char(50), not string.
Automatic Fixed Size Numeric Conversion
Some formats allow for numeric types that have variable width and
precision, and map these types to the fme_decimal(width,precision)
attribute type. Other formats are unable to store variable precision
numeric types and must attempt to map fme_decimal to the correct
size of numeric.
The core does this mapping automatically, by attempting an additional
specialization from the correctly sized fixed numeric FME attribute
type to the destination format attribute types. If the result is
a new fixed precision type, as indicated by a lack of descriptors,
and not equal to the initial result type then that type is preferred.
For example, let's say that an input attribute type of number(15,0)
initially genericized to fme_decimal(15,0) and then specialized to
double in the destination format, by default. The extra fixed size
type conversion would detect the fme_decimal type and identify
fme_int64 as the closest fixed precision numeric type. If
specializing fme_int64 to the destination attribute type resulted
in a type of int64 then this type would be preferred to the initial
double mapping as it is a better fixed numeric size.
Variable and Fixed Strings: Varchar > Char
Also notice that char(width) maps to fme_varchar(width) by default. This is
desirable so that fixed with (char) only formats like ESRI Shape or MapInfo
Tab will map their string values nicely to varchar supporting formats such as
Oracle and other databases without the padding.
For formats that don't have the concept of fixed vs. variable length strings,
you should preferentially map strings to fme_varchar ahead of fme_char. This
means automatic translations to database formats will use varchar columns
instead of char columns, unless the source format knows otherwise.
Width and No Width
Sometimes it is desirable to allow width to be optionally specified for
an attribute type. This can be accomplished using mappings like the
following. Note that it is currently not valid for a Reader to include
"time(x)" on a readSchema feature in this case (PR#58493).
time fme_time \
time(width) fme_time
WORKBENCH_EXPOSABLE_ATTRIBUTES Format Attribute Types
Within Workbench, the WORKBENCH_EXPOSABLE_ATTRIBUTES section lists
all the possible attribute types the format supports.
It is important that this have a full mapping for the generic
FME attribute types, for if Workbench cannot map an FME type (say,
fme_char) bad things will happen (attributes will go missing at
generate time or errors will occur at runtime).
IMPORTANT NOTE: Format specific attributes MUST use the format
attribute types (types on the left hand side of ATTR_TYPE_MAP).
Incorrect use:
WORKBENCH_EXPOSABLE_ATTRIBUTES \
shape_name fme_char(10)
Correct use:
WORKBENCH_EXPOSABLE_ATTRIBUTES \
shape_name char(10)
REQUIRES_FEATURE_TABLES
The REQUIRES_FEATURE_TABLES keyword can be used to specify that a writer
should only ever receive feature tables as input, and should never be
passed a "standard" feature. If this keyword is set to Yes, the core
will accumulate all features into a feature table before passing them to
the writer. In this case, the feature table is constructed as follows:
1. User attributes will be added by parsing the DEF lines and
mapping their types to feature table attribute types.
2. Geometry columns will be added by parsing the DEF lines and
looking for the fme_geometry_columns keyword. If not specified,
the feature table will have a single unnamed geometry. If a
feature table with no geometry is desired, this keyword can be
set to a special value of fme_no_geom.
3. Format attributes will be added from the set of writer exposable
attributes. If only a subset of the exposable attributes is
wanted in the feature table, the FEATURE_TABLE_FORMAT_ATTRS
keyword can be used to specify a subset of the exposable
attributes that should exist in the feature table.
Feature Type Name Constraints
Following metafile directives determine how feature type names are handled.
These are also enforced by Workbench during the course of normal workspace
editing after the workspace has already been generated.
- ASSUMPTIONS:
- ALPHA means english alphabet characters (a-z or A-Z)
- Underscore is the replacement for invalid chars so it is not allowed to
be one of the invalid chars
- . (dot) is the separator between user name and feature type name,
unless it is part of the FEATURE_TYPE_INVALID_CHARS. What this means is
if there is . (dot) in the feature type name and it is not one of the
invalid characters then the FEATURE_TYPE_LENGTH will be from the first .
(dot) position to the end. E.g. For feature type name like "ha
(28.12.09)$0$H_Align - Controls LOC", "ha (28" would be treated as the
prefix or user name and FEATURE_TYPE_LENGTH constraint will be applied on
"12.09)$0$H_Align - Controls LOC". To avoid . (dot) being used as delimiter
between prefix and feature type name, just add FEATURE_TYPE_LENGTH_INCLUDES_PREFIX
directive to the metafile
- FEATURE_TYPE_CASE: This determines whether the attributes should be all
upper case, lower, case, or it doesn't matter. Only affects writer. Possible
values are:
- ANY - Name will be left alone
- UPPER - Name will be forced to upper case including any non-alphabetic
characters. For example accented characters are non-alphabetic but may
have upper and lower case version.
- LOWER - Name will be forced to lower case including any non-alphabetic
characters.
- FIRST_UPPER - First character of the name will be forced to upper case
including any non-alphabetic characters.
- FIRST_LOWER - First character of the name will be forced to lower case
including any non-alphabetic characters.
- UPPER_FIRST_ALPHA - Force name to uppercase and then ensure that first
character is an alphabet (A-Z). Accented characters are not treated as
alphabets.
- LOWER_FIRST_ALPHA - Force name to lowercase and then ensure that first
character is an alphabet (a-z). Accented characters are not treated as
alphabets.
- FIRST_ALPHA - Ensure that first character is an alphabet (A-Z|a-z).
Accented characters are not treated as alphabets.
- ANY_FIRST_NONNUMERIC - Ensure that the first character is not a number,
if it is then put an underscore in front of the name.
- FEATURE_TYPE_LENGTH: This determines the max length of a feature type name
in characters not bytes.
- FEATURE_TYPE_INVALID_CHARS: This determines which characters are invalid
for a feature type name.
- FEATURE_TYPE_RESERVED_WORDS: This determines words that have special
meaning for the format and cannot be used as feature type names. Value of
this directive is space delimited list of words. See sqlreservedwords.fmi
for an example.
- FEATURE_TYPE_LENGTH_INCLUDES_PREFIX: This directive does not take any
value but we provide a dummy value just so that it doesn't mess up where we
expect name value pair when this directive is actually present in the
metafile. Just the presense of this directive indicates that the maximum
length of feature type will be inclusive of the schema or user prefix
usually for database formats. "." is treated as part of the user/schema
name. e.g. A feature type name could be dev.myTables. If this directive is
present then whole of dev.myTables will be used to determine the maximum
allowed length otherwise only myTables will be used for maximum length
calculation.
- WORKBENCH_FEATURE_TYPE_VALIDATION_SCRIPT: Points at a TCL script that
workbench will use to validate feature type names.
Example: In most of our database metafiles we use the following @TCL function
call to apply the feature type name constraints.
@TCL("set no_schema_prefix yes; set invalid_chars_regexp {. / * ; : ! # & - \ " ' $ [ ] }; set max_length 64; source {$(FME_MF_DIR_UNIX)/$(tclFileNameToUse)}")
The above line can be replaced by following metafile directives
FEATURE_TYPE_LENGTH_INCLUDES_PREFIX dummyValue
FEATURE_TYPE_INVALID_CHARS . / * ; : ! # & - \ " ' $ [ ]
FEATURE_TYPE_LENGTH 64
FEATURE_TYPE_CASE ANY
INCLUDE formatReservedWords.fmi
INCLUDE line above is to keep the length of metafile reasonable.
formatReservedWords.fmi will look like
FEATURE_TYPE_RESERVED_WORDS \
ABSOLUTE \
ACCESS \
ACCOUNT \
ACTION \
ACTIVATE \
ADD \
ADMIN \
AFTER \
ALL \
.
.
.
Attribute Name Constraints
There are a few directives that determine how attributes are handled. These
are used by both Workbench and the Universal Translator (UT).
- Assumptions:
- ALPHA means english alphabet characters (a-z or A-Z)
- Underscore is the replacement for invalid chars so it is not allowed to
be one of the invalid char
- ATTRIBUTE_CASE: This determines whether the attributes should be all upper case, lower, case, or it doesn't matter. Only affects writer. Possible values are:
- ANY - Name will be left alone
- UPPER - Name will be forced to upper case including any non-alphabetic
characters. For example accented characters are non-alphabetic but may
have upper and lower case version.
- LOWER - Name will be forced to lower case including any non-alphabetic
characters.
- FIRST_UPPER - First character of the name will be forced to upper case
including any non-alphabetic characters.
- FIRST_LOWER - First character of the name will be forced to lower case
including any non-alphabetic characters.
- UPPER_FIRST_ALPHA - Force name to uppercase and then ensure that first
character is an alphabet (A-Z). Accented characters are not treated as
alphabets.
- LOWER_FIRST_ALPHA - Force name to lowercase and then ensure that first
character is an alphabet (a-z). Accented characters are not treated as
alphabets.
- FIRST_ALPHA - Ensure that first character is an alphabet (A-Z|a-z).
Accented characters are not treated as alphabets.
- ANY_FIRST_NONNUMERIC - Ensure that the first character is not a
number, if it is then put an underscore in front of the name.
- FIRST_LETTER - Ensure that the first character is an Alphabetic
character according to the Unicode definitions. This is a more
expansive version of FIRST_ALPHA (which is limited to Ascii).
- ATTRIBUTE_LENGTH: This determines the max length of an attribute
- ATTRIBUTE_INVALID_CHARS: This determines which characters are invalid in
an attribute name.
- ATTRIBUTE_CASE_IGNORE_LIST: This sets a list of attributes that are not to
be affected by the ATTRIBUTE_CASE directive
- DEST_ILLEGAL_ATTR_LIST: This determines words that have special meaning
for the format and cannot be used as attribute names. Value of this
directive is space delimited list of words. See sqlreservedwords.fmi for an
example. NOTE: This directive is now enforced by workbench.
It's possible for the value of the directive to depend on whether the format
is being read or written. In this case you can check the direction within the
metafile to ensure the correct value is used. But we just said above that
these only affect writers - why bother? The following is an example taken from
geodatabase_sde.fmf:
INCLUDE [if {"$(FMEGEN_DIRECTION)" == "Destination"} { \
puts {ATTRIBUTE_CASE UPPER}; \
puts {ATTRIBUTE_LENGTH 31}; \
} else { \
puts {ATTRIBUTE_CASE ANY}; \
puts {ATTRIBUTE_LENGTH 255}; \
} ]
META_TABLE <TABLENAME> <ACCESSTYPE> [<TABLEVALUE>]+
Old comments (since 2001 at least)
These statements are used to define tables whose values as to be placed in
correlation or definition lines. The idea is that each time the table is
referenced the next table value is used. The table value is incremented as
each feature is processed. This is done to ensure that definition lines
always match corresponding correlation lines. The next value is retreived
from the table using the {FME_TABLE:<tableName>} substitution value.
Currently the <accessType> is stored but not used. The idea would be to
support values such as GROUP_RELATED, SEQUENTIAL, RANDOM, etc. If
GROUP_RELATED is specified then the value returned is associated with the
group. Each group Name is thus associated with an index into the table. This
should be specified if the table value must appear on both an FME mapping
file definition line and a FME correlation line and the values must be in
sync.
New comments (2008)
The common use of this seems to be:
META_TABLE ALLOWED_OUTPUT_FEATURE_TYPES GROUP_RELATED [<FEATURE_TYPES>]+
DEF_LINE_BREAK NO_DEF NO_ATTRIBS
as when it was set to "ATTRIB_CHANGE" the features were duplicated in an
OBJ->OBJ translation and in a translation from Shape->OBJ where the shape
feature type wasn't what was specified in [<FEATURE_TYPES>]+ caused an error
to occur on workspace generation (although that could be related to
WORKBENCH_CANNED_SCHEMA). I also believe that you'll want/need to specify
WORKBENCH_CANNED_SCHEMA <schema file>
The TFS Fixed Schema Plugin is a good example of how this all works.
Single Destination Feature Type Usage
Another common usage is for writers like GENERIC (in dynamic mode), 3DS and
OBJ. In these three cases, the destination dataset has a single workbench
node whose actual feature type and attributes are irrelevant to the writer
at runtime. At file generation time, however, one would want all source
feature types to feed into the single destination type.
There are three points of configuration to set this up in the metafile:
1. A META_TABLE command is added to allow only a single feature type to be emitted
2. A canned schema is used to define the one feature type that will be emitted
3. The DEF_LINE_BREAK is set so that all input is connected. The internals of FME
will detect the case where META_TABLE is limiting the output to a single table,
and will join all input feature types to the one output feature type at generation
time.
In the metafile for 3DS, these three lines look like the following. The
order is not significant, but they must be located in the appropriate
sections of the metafile. The "NO_ATTRIBS" prevents the user from adding
attributes to the destination feature type.
META_TABLE ALLOWED_OUTPUT_FEATURE_TYPES GROUP_RELATED 3DS_ELEMENT
WORKBENCH_CANNED_SCHEMA 3ds.sch
DEF_LINE_BREAK ATTRIB_CHANGE NO_ATTRIBS
This will generate workspaces with all input feature types joined to a
destination feature type named "3DS_ELEMENT". This type is defined in
schemas/3ds.sch as:
FEATURE_DEF 3DS_ELEMENT \
fme_geometry{0} 3ds_mesh \
fme_geomattr{0}.3ds_diffuse_color string \
fme_geomattr{0}.3ds_specular_color string \
fme_geomattr{0}.3ds_ambient_color string \
fme_geomattr{0}.3ds_material string \
fme_geomattr{0}.3ds_texture_image string \
fme_geomattr{0}.3ds_texture_map_blur string \
fme_geomattr{0}.3ds_texture_map_scale string \
fme_geomattr{0}.3ds_texture_map_offset string \
fme_geomattr{0}.3ds_texture_map_rotation string \
fme_geomattr{0}.3ds_texture_map_mirror string \
fme_geomattr{0}.3ds_texture_map_intensity string
Correlation Line Directives
CORR_LINE_TEMPLATE defines the format of the correlation line. It is of the form:
CORR_LINE_TEMPLATE <groupName> [<def_attr> <def_attr_value>]*
For formats with only one format-specific type, this is typically:
CORR_LINE_TEMPLATE {FME_GEN_GROUP_NAME}
For formats with multiple format-specific types, this is typically:
CORR_LINE_TEMPLATE {FME_GEN_GROUP_NAME} <format-specific-type-attribute> {FME_GEN_GEOMETRY}
The second form is needed on both the source and destination sides whenever
the format has more than one format-specific type. If you leave this out on
the reader side, you can't successfully convert to a format that splits the
data up by geometry type (e.g., Shape). If you leave this out on the writer
side, the writer won't get the format-specific type. These bad symptoms only
occur if you are using the Universal Translator or generated mapping files.
Therefore, it is good practice to test each format by converting the alltypes
MIF to the format, and then converting the format to Shape, all via the
Universal Translator.
FORMAT_TYPE
The type of format (BUILTIN, DYNAMIC, WRAPID, JAVA, PYTHON, TRANSFORMFACT, DEPRECATED).
This line will prevent FME from iterating through each type when creating the
reader or writer. This allows more specific errors to appear in the logfile,
and prevents misleading log messages from appearing should FME fail to create
this reader or writer.
Note: The existence of the keywords DYNAMIC_LIBRARY* implies FORMAT_TYPE DYNAMIC
FORMAT_TYPE_READ
Same as FORMAT_TYPE, except specific for the reader.
FORMAT_TYPE_WRITE
Same as FORMAT_TYPE, except specific for the writer.
If the format type is DEPRECATED, the reader and writer creators will log
that and suggest a preferred format as specified by the following:
DEPRECATED_FORMAT_ALTERNATIVE <FORMAT_SHORT_NAME>
DYNAMIC_LIBRARY
Name of the dynamic library (minus the extension) to use for the reader/writer
relative to FME_HOME/plugins/ and FME_HOME/ (in that order). This metafile
directive:
- allows a single dynamic library to be shared by multiple formats
- for example, AutoCAD DWG/DXF and AutoCAD DWF both share the same
dynamic library
- provides a specific folder for storing 3rd party files specific to the
plug-in. For example, 3rd party C++ dynamic libraries can be stored here
and will automatically get loaded when needed (note: .NET DLLs stored here
will not automatically get loaded at runtime and therefore should be
registered with the GAC or stored in the FME_HOME folder).
- helps minimize the number of files/folders located at the FME_HOME root
location
If you would like to store your FME plug-in dynamic library in a subfolder within
the FME_HOME/plugins directory (like PDF does), then the metafile directive will
be:
DYNAMIC_LIBRARY <folder name>/<dynamic library name minus file extension>
DYNAMIC_LIBRARY_READ
Same as DYNAMIC_LIBRARY, except specific for the reader.
DYNAMIC_LIBRARY_WRITE
Same as DYNAMIC_LIBRARY, except specific for the writer.
Re-using entry points from an existing plugin
Note that if you are adding a new reader (or writer), which piggybacks off
an existing format (i.e. gdal or xml), and the existing format which you're
piggybacking off of contains a the writer dynamic library entry point
FME_createWriter() (or FME_createReader()) then you'll need to add the
following line in the metafile to inform our FORMAT reader that you're only
a reader:
DYNAMIC_LIBRARY_WRITE __NOWRITER__
If you're a writer only, then you'll need to add the line:
DYNAMIC_LIBRARY_READ __NOREADER__
The purpose of this is so that the FORMAT reader will provide the value NO to the attribute CAN_CREATE_[READER|WRITER].
PLUGIN_NAME
The actual plugin name for python and class name for Java plugins.
This specify the module to load.
PLUGIN_NAME_READ
Same as PLUGIN_NAME, except specific for the reader.
PLUGIN_NAME_WRITE
Same as PLUGIN_NAME, except specific for the writer.
FME_PYTHON_MINIMUM_VERSION
To specify the minimum python version inside the metafile.
FME_PYTHON_MINIMUM_VERSION 2.6
FME_PYTHON_PATH
Specify a python path inside the metafile.
FME_PYTHON_PATH "$(FME_HOME)../foundation/formats/ogdi/"
COORDINATE_SYSTEM_GRANULARITY
This directive defines the lowest level of coordinate system granularity
supported by a format. Valid values are {DATASET, FEATURE_TYPE, FEATURE}.
Although this directive can be added anywhere in the metafile but it is
strongly suggested that this directive be used within the DESTINATION_PREAMBLE
section of the metafile to maintain backwards compatibility. By adding the
directive in DESTINATION_PREAMBLE section will ensure that the value of this
directive will only apply to newly generated workspaces. Existing workspaces
should work the same way as before this change. If adding a new format you can
choose to add it any where in the metafile. Here is bit of explanation on what
the directive value means.
- DATASET: This is the old behaviour where all features sent to a writer
will have the same coordinate system. The very FIRST feature with
coordinate system (it may not be the very first feature sent to the
writer) was used as the reference coordinate system for all the subsequent
features. If any feature had a different coordinate system it would then
be reprojected to the reference coordinate system. If you don't specify
the COORDINATE_SYSTEM_GRANULARITY directive, this is assumed by default.
But if you do specify this directive explicity in the metafile then a
slightly modified approach will be taken to get the reference coordinate
system. Instead of using the first feature with coordinate system, FME
would use the first feature's coordinate system as the reference
coordinate system. This mean that if the first feature doesn't have any
coordinate system specified then subsequent features that may have a
coordinate system will never be reprojected
- FEATURE_TYPE: This will ensure that the same coordinate system will be set
for a given feature type. Again the coordinate system on the first feature
for a given feature type will be used as the reference coordinate system.
All other subsequent features will be re-projected to the reference
coordinate system if it is different and reference coordinate system is not
empty. Most database formats will be in this category.
- FEATURE: This will will ensure that FME core will never touch the
coordinate system on the feature, it will be up to the writer to handle per
feature coordinate systems. Most raster writers will be in this category.
IMPLEMENTS_FTR
This directive defines if the reader has implemented support for FEATURE_TYPES
directive when reading schema and data features. What this means is if a reader
filters the features based on the FEATURE_TYPE directive natively, then reader
core (STFReaderImp) doesn't have to do any filtering based on FEATURE_TYPES
directive. WMS reader uses this directive because the FEATURE_TYPES list
doesn't match with the features that are returned because of the hierarchical
nature of the feature type naming scheme. E.g.
$(FMEGEN_SOURCE_KEYWORD)_IMPLEMENTS_FTR yes
$(FMEGEN_DESTINATION_KEYWORD)_MULTI_WRITER_DEFER_WRITING
Specifies instructions to the multi-writer for a format writer, to help it
make the decision to create the format writer as resident versus deferred.
The default without this specified behaves like a value of 'ANY', where the
multi-writer may choose to defer or not as it has before.
ALWAYS is the only currently supported setting, which indicates that format
writer instances should always be deferred, and never made resident. This
can be useful to avoid access conflicts between readers and writers that can
occur during resident writing.
** Metafile Directives Specific to Workbench **
_WORKBENCH
This is a source and destination macro set in the control file object, which is
accessible in metafiles with values 'yes' or 'no'. It is intended to indicate
two things: that the metafile usage is currently for workbnech, and that the
metafile usage is currently for generate.
WORKBENCH_DEFLINE_PARMS
This section gives more information about the DEF line parameters in the
DEF_LINE_TEMPLATE. Each parameter can supply a GUI line and a default value.
This defines how the GUI will appear under the 'Parameters' tab in the feature
type properties dialog.
Warning: Traditional wisdom was that settings needed only in Workbench should
be here and could be omitted from the DEF_LINE_TEMPLATE. This isn't true for
writers! Parameters missing from DEF_LINE_TEMPLATE will be ignored during
dynamic writing.
GUI Type: DISABLE
A type of DISABLE means that the parameter is a YES/NO choice and other
parameters will be enabled/disabled in response to changes to this parameter.
When the DISABLE parameter is NO, parameters with the DISABLED_ONLY modifier
are disabled. When the DISABLE parameter is YES, parameters without the
DISABLED_ONLY modifier are disabled.
Parameters with the ALWAYS_ENABLED modifier are not affected by DISABLE parameters.
GUI Modifier: FORCE_DEFS
Based on apps/workbench/controlDefs.cpp, it appears that if any Workbench parameter has the FORCE_DEFS modifier all attributes will be included in the DEF line regardless of whether the format "writes out def lines" (whatever that means).
// get if this guy is a "disabled database" record node
// this occurs with certain formats if the user selects that the
// database table corresponding to this feature type already exists.
// if this is true, then the defline parameter for this guy should
// not be written out
virtual bool DisabledDeflineParms();
WORKBENCH_EXPOSABLE_ATTRIBUTES
This gives a list of all the attributes that Workbench will show under the
'Format Attributes' tab in the feature type properties dialog, and the
attributes' types.
WORKBENCH_ADDITIONAL_ATTRIBUTES
NOTE: This keyword has been deprecated.
Use [FORMAT_PARAMETER WRITER_AUTO_EXPOSE] for automatically exposing (adding
to destination feature types) attributes on destination feature types.
This gives a list of attributes that will, by default, be added to destination
feature type. This is used in combination with the
WORKBENCH_EXPOSABLE_ATTRIBUTES tag to automatically expose format attributes for
destination feature types. It has the same syntax as
WORKBENCH_EXPOSABLE_ATTRIBUTES.
WORKBENCH_CANNED_SCHEMA
This points to a schema file (the .sch files in install/schemas) defining all
the canned schemas that will end up as destination feature types for this
format. This directive is only applicable to writers. See the GXF metafile for
an example.
Alternatively instead of a hardcoded .sch file being specified, this directive
can also be given the value READ_AS_SOURCE which indicates that the writer
should use the reader readSchema call to determine its schemas. This can be used
in cases where there are several fixed schemas to be chosen from dynamically
based on use input. To be clear the source reader is not used as in the variable
schema cases, rather the destination format uses the same format reader to read
its own schema accurately. I.e. Shape to ADAC would use the ADAC read to read
the known schemas for the ADAC format from a well known .xsd file shipped with
FME. This is a similar but more dynamic way of defining a set fixed schemas for
a writer without making static .sch files. Another example is OpenSpirit formats
where the user chooses from several sets of fixed schema data models which we do
not want to define in .sch files for maintenance reasons.
When this directive is specified, then the directive
WORKBENCH_NO_FEATURE_TYPE_FANOUT must also be specified because fan-out does not
make sense in a fixed schema format.
WORKBENCH_NO_FEATURE_TYPE_FANOUT
By default, Workbench assumes your format will support feature type fanout.
Your format will get this for free if it is a file-based format. If it is
directory-based, or a database format, your format will require special code
(see: Feature Type Fanout). If you have a directory or database format that
does not support feature-type fanout, put this stand-alone directive in your
metafile. Workbench will then disable the fanout option in the feature type
properties dialog.
WORKBENCH_ATTR_INDEX_TYPES
This specifies the types of indexes that attributes on this format can have. If
this is specified, an extra column is provided in the feature type properties
dialog for this format, allowing the user to select indexes for specific
columns. The syntax looks like this:
WORKBENCH_ATTR_INDEX_TYPES <type1> <type2> <type3>
If the user selects an index for an attribute, it appears after the
attribute's type on the feature type's DEF line, separated by a comma. For
instance:
SPUD_DEF MyTable \
column_one varchar(255) \
column_two varchar(10),indexed
FORMAT_PARAMETER
This is a generic way of specifying reader/writer parameters for use in
Workbench. The way you specify such a parameter is like this:
FORMAT_PARAMETER <key> <value>
The key is completely arbitary. Using FORMAT_PARAMETER, instead of making a
whole new directive (as the ones above), allows you to make new metafile
parameters instantly, without having to modify about 10 C++ classes. Any
parameters specified in this way will automatically be stored in workspaces
and be usable in Workbench. It is anticipated that most new reader/writer
configuration directives will be done this way.
FORMAT_PARAMETER AUTO_PUBLISH
Specifies which reader/writer directives should be published when added to a
workspace. If this is not specified, the default behaviour is to publish the
source and destination dataset parameter.
NOTE : If this is used in a metafile to auto publish some parameter(s) then
ensure that SourceDataset and DestDataset are also included in the list for
auto publishing. Syntax:
FORMAT_PARAMETER AUTO_PUBLISH <macro1> <macro2> ...
<macro1> <macro2> are the macro names used in in the reader/writer preambles.
These are the macros that will be published.
FORMAT_PARAMETER ADVANCED_PARMS
Specifies which reader/writer directives should be shown under the 'Advanced'
branch, instead of in the main 'Parameters' branch in the navigation tree.
Syntax:
FORMAT_PARAMETER ADVANCED_PARMS <macro1> <macro2> ...
<macro1> <macro2> are the macro names used in in the reader/writer preambles.
These are the macros that will be shown under 'Advanced'. Each format can only
have one ADVANCED_PARMS list, so the reader and writer have to share a list.
Note: There is a more complete syntax for this for two reasons.
When using DEFAULT_VALUE <name> lines, the name is mangled into a macro by the
time the ADVANCED_PARMS list is parsed. Thus the list must specify the mangled
macro name in a generic fashion if the list it is to work.
The advanced parameters get parsed once for the source format and once for the
destination format. If you don't limit when each type of macros get used, you will
end up with something like destination macros in the source advanced parameters list.
Thus this syntax is better. Syntax:
INCLUDE [if { ("$(FMEGEN_DIRECTION)" == "Source") } { \
set line "FORMAT_PARAMETER ADVANCED_PARMS"; \
append line { $(FMEGEN_SOURCE_FORMAT)_IN_<source_macro_name>}; \
puts $line; \
} else { \
set line "FORMAT_PARAMETER ADVANCED_PARMS"; \
append line { $(FMEGEN_DESTINATION_FORMAT)_OUT_<destination_macro_name>}; \
puts $line; \
}]
FORMAT_PARAMETER SEARCH_ENVELOPE_PARMS
Specifies which additional reader/writer directives should be shown under the 'Search Envelope'
branch, instead of in the main 'Parameters' branch in the navigation tree. Directives whose
name contains "SEARCH_ENVELOPE" or "CLIP_TO_ENVELOPE" are automatically added to
'Search Envelope' category, so there is no need to list those directives here.
Syntax:
FORMAT_PARAMETER SEARCH_ENVELOPE_PARMS <macro1> <macro2> ...
For information on how to add in the metafile and how to name the directives, see
details ADVANCED_PARMS parameter above.
FORMAT_PARAMETER READER_AUTO_EXPOSE
Specifies a list of attribute name and attribute type pairs, to be automatically exposed on source feature types.
Syntax:
FORMAT_PARAMETER READER_AUTO_EXPOSE <attrName1> <attrType1> <attrName2> <attrType2> ...
FORMAT_PARAMETER WRITER_AUTO_EXPOSE
Specifies a list of attribute name and attribute type pairs, to be automatically exposed on destination feature types.
Syntax:
FORMAT_PARAMETER WRITER_AUTO_EXPOSE <attrName1> <attrType1> <attrName2> <attrType2> ...
FORMAT_PARAMETER ALLOW_DATASET_CONFLICT
Specifies workbench to ignore dataset conflict warning, if a format can read
from one dataset and write back to same dataset. If this is not specified, the
default behaviour is to is to warn user if source and destination datasets are
same. Syntax:
FORMAT_PARAMETER ALLOW_DATASET_CONFLICT [YES|NO]
FORMAT_PARAMETER AUTO_FANOUT_ATTRIBUTE
Specifies workbench to use the "attributeName" specified as the attribute to
use for setting the fanout property of destination dataset during workspace
generation, adding new dataset or adding new feature type. For now the
attribute type is assumed to be char(50), although I believe it doesn't matter
anyways. Syntax:
FORMAT_PARAMETER AUTO_FANOUT_ATTRIBUTE <attributeName>
FORMAT_PARAMETER SUPPORTS_ESTABLISHED_CACHE
<To be written>
FORMAT_PARAMETER DEFAULT_READER
Specifies workbench to use this reader when invoking the inspection application for visualizing the Writer dataset.
FORMAT_PARAMETER DEFAULT_READER <reader short name>
For example: In the Excel writer (XLSXW) metafile add this to use a different reader - FORMAT_PARAMETER DEFAULT_READER XLSXR
FORMAT_PARAMETER [FORMAT|READER|WRITER]_REPLACED_BY <format short name>
Specifies that this format has now been deprecated and replaced by new
format whose short name is provided as a value of the parameter. This
is currently being used to determine which format to use when updating
deprecated formats during "update dataset" operation
For example: FORMAT_PARAMETER FORMAT_REPLACED_BY CSV2
FORMAT_PARAMETER NETWORK_AUTHENTICATION
Specifies fme/workbench whether or not to automatically add network
authentication parameters to the reader settings box dialog. If this keyword
is not specified fme/workbench will defaults its value to be YES. If
this keyword is set to YES and the source dataset is a URL, fme/workbench
will add network authentication parameters to the settings box. If this
keyword is set to ALWAYS, the authentication parameters will be added
for all datasets. If a reader supports urls for its dataset but want
to handle network authentication using their own methods or it doesn't
apply to them then add this keyword with value of NO.
FORMAT_PARAMETER NETWORK_AUTHENTICATION [YES|NO|ALWAYS]
FORMAT_PARAMETER NETWORK_PROXY
Specifies fme/workbench whether or not to automatically add network proxy
parameters to the reader settings box dialog. If this keyword is not specified
fme/workbench will defaults its value to be NO. If a reader wants to prompt user
for proxy information then add this parameter in the metafile with a value of YES.
FORMAT_PARAMETER NETWORK_PROXY [YES|NO]
FORMAT_PARAMETER READER_FEATURE_TYPE_EDITING
Specifies fme/workbench whether or not to allow editing of reader feature
type. Value of this metafile keyword used in conjunction with the global
setting "Tools|FME Otions...|Workbench|Allow reader feature type editing". If
this keyword is not specified fme/workbench will defaults its value to be YES.
If global setting for "Allow reader feature type editing" is set to "NO" then
this keyword does not apply. If a reader cannot fully support/allow editing of
reader feature type then add this keyword with value of NO. E.g. CSV
FORMAT_PARAMETER READER_FEATURE_TYPE_EDITING [YES|NO]
FORMAT_PARAMETER WRITER_FEATURE_TYPE_EDITING
Specifies fme/workbench whether or not to allow editing of writer
feature type. If this keyword is not specified fme/workbench will
defaults its value to be YES. Use it for writer's with fixed schema
that would like to show the user attributes but not allow editing
of those attributes. Specify this parameter only if you want to
disable writer feature type editing.
FORMAT_PARAMETER WRITER_FEATURE_TYPE_EDITING [YES|NO]
FORMAT_PARAMETER FEATURE_TYPE_NAME
Specifies the name of the feature type for a given format. E.g. For AutoCAD
it would be "Layer" and Oracle it is "Table"
FORMAT_PARAMETER FEATURE_TYPE_NAME <format specific name>
FORMAT_PARAMETER FEATURE_TYPE_DEFAULT_NAME
Specifies the default name to use when in Workbench a new feature type
is added E.g. For AutoCAD
it would be "Layer1" and Oracle it is "Table1"
FORMAT_PARAMETER FEATURE_TYPE_DEFAULT_NAME <format specific name>
FORMAT_PARAMETER DATASET_NAME
Specifies the dataset type name for a given format. E.g. For AutoCAD
it would be "dwg file" and Oracle it is "Database"
FORMAT_PARAMETER DATASET_NAME <format specific dataset name>
FORMAT_PARAMETER READER_DATASET_HINT
Specifies a short description for a reader dataset field. E.g. For AutoCAD
it could be "Select the AutoCAD DWG file(s)" and Oracle it could be
"Specify the Oracle Spatial Relational Service". This text is used as
tooltip or a gray text for the dataset field in add reader dialog
FORMAT_PARAMETER READER_DATASET_HINT <format specific reader dataset hint>
FORMAT_PARAMETER WRITER_DATASET_HINT
Specifies a short description for a writer dataset field. E.g. For AutoCAD
it could be "Specify a name for the AutoCAD DWG file" and Oracle it could be
"Specify the Oracle Spatial Relational Service". This text is used as
tooltip or a gray text for the dataset field in add writer dialog
FORMAT_PARAMETER WRITER_DATASET_HINT <format specific writer dataset hint>
FORMAT_PARAMETER WRITER_ADD_FEATURE_TYPE_DEFAULT
Specifies one of {copy_from_reader, import, import_only, define_new} the
option to select by default when adding a new writer. If "import_only" is
specified then all the other options on the add writer dialog will be
disabled.
FORMAT_PARAMETER WRITER_ADD_FEATURE_TYPE_DEFAULT [none|automatic|manual|copy_from_reader|import|dynamic]
FORMAT_PARAMETER DEFAULT_ATTR_TYPE
Specifies a format attribute type to use when adding a new writer
attribute in workbench. This type will be used when mapping from
reader to writer attribute type cannot be found or when the attribute
type is not known in general
FORMAT_PARAMETER DEFAULT_ATTR_TYPE <format specific attribute type>
E.g.
FORMAT_PARAMETER DEFAULT_ATTR_TYPE auto
FORMAT_PARAMETER DEFAULT_GEOMETRY_TYPE
Specifies a format specific geometry type to use when adding a new
writer in workbench. This will be also used when mapping geometry type
from reader over to writer feature type in cases where reader support
multiple geometries and writer doesn't. Once such scenario is when
copying/duplicating writer feature type from existing reader feature
types
FORMAT_PARAMETER DEFAULT_GEOMETRY_TYPE <format specific geometry type>
E.g.
FORMAT_PARAMETER DEFAULT_GEOMETRY_TYPE shape_first_feature
FORMAT_PARAMETER SUPPORTS_SCHEMA_IN_FEATURE_TYPE_NAME
Specifies whether the feature type names should be prefixed with the schema.
Setting to YES will allow editing of the Table Qualifier parameter in the
feature type properties dialogs. If not present, then the parameter is
provided iff the format's DATASET_TYPE is DATABASE in 'formats.db'.
FORMAT_PARAMETER SUPPORTS_SCHEMA_IN_FEATURE_TYPE_NAME [YES|NO]
FORMAT_PARAMETER USER_ATTRIBUTES_COLUMNS
Allows for the renaming of the columns on the user attributes tab of the writer
feature type properties. Specifies a mapping of a old value to new value.
Optionally a new tooltip string can be provided as well, if one is not provided
the new name will be used as the tooltip.
FORMAT_PARAMETER USER_ATTRIBUTES_COLUMNS <Old Column Name>,<New Column Name>[,<New Tooltip>]%...
E.g.
FORMAT_PARAMETER USER_ATTRIBUTES_COLUMNS Width,Cell Width%Precision,Formatting
FORMAT_PARAMETER ATTRIBUTE_READING [ALL|DEFLINE|DEFLINE_ATTRS]
Indicates how readers handle attributes to read. If not specified,
then ALL will be assumed.
ALL - All user attributes are always read. In this case,
the reader always reads all the attributes that exist
from the actual data runtime. In workbench, all the
attributes in the User Attributes table will always be
put onto the DEFLINE.
DEFLINE - Only specified attributes on the DEFLINEs are read. In
this case, the reader uses attributes specified on the
DEFLINE to determine what attributes to read. A special
defline attribute, 'fme_attribute_reading', will be
added to the DEFLINE with a value of 'all' or 'defined' where
'all' means read all attributes at runtime regardless of
what's defined on the DEFLINE and 'defined' means read only
the attributes that are specified on the DEFLINE.
DEFLINE_ATTRS - (Deprecated) Prefer DEFLINE for new readers.
Only specified attributes on the DEFLINEs are read. In
this case, the reader uses attributes specified on the
DEFLINE to determine what attributes to read. No attributes
on the DEFLINE signals that the reader should read whatever
exists on the feature type at runtime.
When ATTRIBUTE_READING is specified, a DEFAULT_MACRO will be added to
the generated mapping file with the same value. For example,
FORMAT_PARAMETER ATTRIBUTE_READING DEFLINE_ATTRS
will generate the following macro in the mapping file:
DEFAULT_MACRO <reader_keyword>_ATTRIBUTE_READING DEFLINE_ATTRS
When changing the behaviour of a reader, additionally add the
ATTRIBUTE_READING_HISTORIC format parameter so that pre-existing
workspaces continue to work as before.
FORMAT_PARAMETER ATTRIBUTE_READING_HISTORIC [ALL|DEFLINE_ATTRS]
Allows a format to change its behaviour for ATTRIBUTE_READING, while
maintaining behaviour for pre-existing workspaces. Add this whenever
changing the original behaviour of a format that did not always have
ATTRIBUTE_READING defined.
Values are the same and have same meaing as ATTRIBUTE_READING.
FORMAT_PARAMETER PARAMS_TO_NOT_PROPAGATE_ON_INSPECT <Param1> <Param2> ...
Specifies a list of parameters that should not automatically be used
when passing parameters on inspect. For example, reader SQL statements
may modify the dataset and should not be used by default in a data
inspection context.
E.g.
FORMAT_PARAMETER PARAMS_TO_NOT_PROPAGATE_ON_INSPECT BEGIN_SQL{0} END_SQL{0}
FORMAT_PARAMETER SQL_EXECUTE_DIRECTIVES (ALL|INCLUDE|EXCLUDE)[:<param1>[%<param>]*]
Specifies which directives should be passed to the reader when executing
an fme_sql_execute constraint.
ALL - Specifies that all reader directives will be included
INCLUDE - Specifies that only the listed directives are included; the
rest will be disregarded.
EXCLUDE - Specifies the listed directives will be disregarded; the
rest will be included.
E.g.
FORMAT_PARAMETER SQL_EXECUTE_PARAMS INCLUDE:NAMED_CONNECTION%TABLELIST
FORMAT_PARAMETER SUPPORTS_ORDERED_FEATURE_TYPE_READING yes|no
This parameter is used in conjunction with the IDLIST directive and is
applicable only when using the FMEObjects API. It specifies whether
the reader supports reading feature types in the order feature types are
specified in the the IDLIST directive. This is essentially an opt-in
feature for readers, allowing FMEObjects clients, such as FeatureReader,
to provide additional means to order the feature types.
E.g.
FORMAT_PARAMETER SUPPORTS_ORDERED_FEATURE_TYPE_READING yes
**Miscellaneous**
Getting Classic-style Settings Box Values to Work in FME Objects Applications
(e.g. Data Inspector) (Internal Only)
IMPORTANT NOTE: All the information in this section pertains to custom settings
boxes (where lines like "GUI FORM" and "GUI CONTROLLER" are used) and not the
new way of specifying settings boxes through the metafile.
FME Objects requires an explicit mapping of macro names used in
metafiles to reader/writer directives. This is done by adding the
following lines into fmeobjdefs.h:
"<format directive",
"@<macro name>@",
Here's a real-life example:
"READ_3D_POLYGON_AS_FACE",
"@_ORACLE8I_Read3DPolygonAsFace@",
If you set the FME_DEBUG environment variable to DUMP_CONFIG, you should
be able to see the effect of this - see documentation on the FME_DEBUG
environment variable for more information.
Another Method
Around 2005, still before the advent of metafile generated settings
boxes, another method was developed that prevented developers from
having to update the fmeobjdefs.h file. When the macro name adheres to
the following convention:
<format short name>_<IN|OUT>_<directive>
then no explicit mapping in fmeobjdefs.h is required. For example, the
MySQL reader has a keyword called PASSWORD. Following the convention,
the macro name is therefore MYSQL_IN_PASSWORD. The MySQL writer also
has the same keyword, but because it is a writer keyword, the macro
will contain OUT instead of IN - resulting in MYSQL_OUT_PASSWORD.
Value Matching
Matching between different attribute values must also be handled. A good example
of this is the matching of colors from one system to another. Clearly it is not
possible to explicitly match colors since a system like MapInfo specifies 16
million different colors. Instead it is proposed that the FME support 64
different colors. 4 bits for blue, 4 for red, and 4 for green. The mapping of
colors from other systems to these is accomplished by dividing the colors
appropriately. For example 24 bit color is converted to FME color values as
follows:
MapInfo Green Component / 16
MapInfo Red Component / 16
MapInfo Blue Component /16
Hopefully this mapping can be accomplished using @Evaluate(). If not then
another alternative will have to be done. Of course systems which have 64 or
fewer defined colors can be directly mapped. It is important to note that we
cannot merely divide the entire color value by a single value to get the same
effect as the colors at the bottom end of the value will be unfairly
discriminated against.
Predfined Macros
A number of predefined macros are required in order to specify information
properly. These macros are defined and can be referenced throughout FME meta
files.
$(FMEGEN_SOURCE_KEYWORD):
This macro expands to the source keyword which will be used throughout the resulting mapping file.
$(FMEGEN_DESTINATION_KEYWORD):
This macro expands to the source keyword which will be used throughout the resulting mapping file.
Substitution Values
These values (aside from being poorly named) are used to specify that you want
the current group name to be substituted into a definition line or correlation
line. It is expected that these will be used primarily for the specification of
definition lines and correlation lines.
{attrName}: This is used to extract an attribute value from the schema
generation feature and store it within a definition line or correlation
line. This is currently designed for use on a definition line and removes
the referenced attribute from the feature once it is substituted. See the
SDEdefs.txt file for an example of how to use this functionality. It is used
in the DEF_LINE_TEMPLATE statement.
{FME_GEN_GROUP_NAME} This macro contains the current name of the group as
defined by the current source feature which is being used to drive the
mapping file generation. This value is useful for defining definition and
correlation lines within a meta file.
{FME_GEN_GROUP_BASE} This macros is the same value as the GEN_GROUP_NAME above
except that if the group name contains any file extension names then the file
extension is removed. For example in MOEP 83d088d.bin would be output as
83d088d for {FME_GEN_GROUP_BASE} and 83d088d.bin for {FME_GEN_GROUP_NAME}
{FME_GEN_GROUP_NUMBER} This is not supported as the same functionality can
be performed using the {FME_TABLE:<meta table name>} directive with more
flexibility. See igdsdefs.txt for example of how this is done.
{FME_GEN_GEOMETRY} This value contains the current geometry value of the
current definition line in the form consistent with the format which the
value is specified.
{FME_TABLE:<meta table name>} This value is substituted by the current value in
the meta table identified by <meta table name>.
**Q & A**
*What is the purpose of the GENERICIZE_GEOMTYPE_ATTR_NAME line in a
metafile? Nothing seems to break when I commented that line out of the
sample, as long as I have the generic format attribute listed on the
CORR_LINE_TEMPLATE line.*
That directive was only put there so that WHEN the CORR_LINE_TEMPLATE does not
list it (which in many situations it does not) there is a way of specifying
the attribute. So there's no harm in asking for it always to be listed, it
decouples the two situations and makes things cleaner likely. the GENERICIZE..
one was added later, much later, and I only put it into metafiles where the
CORR LINE thing didn't have what I needed (so the FME basically can get the
GENERICIZE attribute thing from the corr line, preventing a need to update all
existing metafiles).
For the plugin builder, you should always specify the
GENERICIZE_GEOMTYPE_ATTR_NAME line.
*What is the difference between comments that begin with '!' and '#'?*
-'!' comments are metafile comments, '#' comments are comments that are
directly inserted into the generated mapping file. -'#' are only used
used in preamble declarations.
*What is the meaning of '__'?*
This is used to insert a return line in the generated mapping file. The
characters are only used in preamble declarations.
*What is the purpose of preambles in a metafile?*
Preambles are a template of comments and directives that are used when
mapping files are generated. There are three types of preambles:
preamble, source_preamble, and destination_preamble. Used to define
comments and directives that are included mapping files generated for
source/destination, source, and destination metafiles (respectively).
There are also two more preambles for workbench-only things, see the
SHAPE metafile for an example. Stuff within the workbench preamble
sections is only inserted when writing a workspace. The reason we have a
seperate section is so that you can only put the essential GUI prompts
in the regular metafile sections, so that when people do a drag and drop
translation it will only prompt for essential things, but you can expose
ALL the format's parameters in workbench, which has a nice tree view for
parameter editing.
*What is a metafile for?*
* Mapping file generation
* Generification
*Is it possible to add a line to a metafile, so that the
FME_MINIMUM_BUILD is added to the generated mapping file?*
Yes, you can put the statement into the "PREAMBLE" section of your
metafile. Example:
PREAMBLE
FME_MINIMUM_BUILD 999
END_PREAMBLE
*What are essential GEOM_MAP lines in a metafile?*
I think that this depends on your format. If you're thinking minimal,
you probably should look to the textline format. Here are it's lines
* GEOM_MAP text_line_none fme_no_geom
* GEOM_MAP text_line_none fme_point
* GEOM_MAP text_line_none fme_line
* GEOM_MAP text_line_none fme_polygon
* GEOM_MAP text_line_none fme_text
* GEOM_MAP text_line_none fme_ellipse
* GEOM_MAP text_line_none fme_arc
* GEOM_MAP text_line_none fme_rectangle
* GEOM_MAP text_line_none fme_rounded_rectangle
This maps all fme geometries to none, and maps the text_line_none
geometry to 'fme_no_geom'
*Are ATTR_MAP and GEOM_MAP defining mappings between types? Then what do
the many to many mappings mean?*
Order matters. The first matching element from the relevant side is
matched to its opposing side. So an fme_geometry is matched to
text_line_none but text_line_none is mapped only to fme_no_geom.
*What other essential lines are there in metafiles? (i.e. that all
formats should have*
You'll need to look to Workbench_and_Metafiles, but here are three that can be
easily overlooked
* ATTRIBUTE_CASE ANY
* ATTRIBUTE_LENGTH some_number. If you don't have a limit on your
reader/writer, use the value 255. You will find other examples, but
use 255.
* ATTRIBUTE_INVALID_CHARS