An FME translation is controlled by a series of rules specifying the guidelines and transformations of a data translation. The mapping file drives the operation of all FME modules during a data translation. The typical format of the FME mapping file: ****************************************** * Reader and Writer Configuration * ****************************************** * Feature Factory Pipeline configuration * *----------------------------------------* * Feature Factory 1 * * Feature Factory 2 * * | * * \_/ * * Feature Factory n * ****************************************** * Transforamtion Module Configuration * *----------------------------------------* * Feature Definition Statements * * Feature Correlation Rules * ****************************************** Although the order of the major sections of a mapping file is not significant, typically the configuration of readers and writers is placed at the top. Next, the factory definitions, if any, are presented. Finally, the definitions of the features in the input and output systems are given with the rules for transforming them from one system to the other. Specifying Reader and Writer Format The two most important FME keywords specify the reader and writer modules to be used during an FME session. The syntax of these keywords are: READER_TYPE WRITER_TYPE Any valid reader or writer module may be named following these keywords. When the FME session is started, the mapping file is scanned for these two keywords and the appropriate reader and writer are created. The mapping file fragment below creates a SAIF reader and a Shape writer: READER_TYPE SAIF WRITER_TYPE SHAPE Specifying Reader and Writer Parameters When a reader and writer are created, they scan the mapping file for additional parameters to configure their operation. All reader and writer parameters are labeled with a two-part keyword. The prefix of the keyword is the current keyword associated with the reader or writer. The suffix identifies the configuration option. The general form of these parameters is: _ _ Note: The above examples should appear as a single continuous line. By default, the is the same as the READER_TYPE setting, and the is the same as the WRITER_TYPE setting. For example, if the READER_TYPE and WRITER_TYPE are configured as: READER_TYPE SAIF WRITER_TYPE SHAPE then by default the is SAIF and the is SHAPE. When the SAIF reader is created, it searches the mapping file for parameters prefixed with SAIF_ and the Shape writer searches for parameters prefixed with SHAPE_. Transformation specifications determine the mapping between source features and destination features. The starts every source line of a transformation specification and the starts every destination line. In the example above, the transformation module scans the file for pairs of lines starting with SAIF and SHAPE, with the SAIF lines being the source portion of the transformation specification and the SHAPE lines being the destination portion. The default values for the and may be overridden by specifying values for READER_KEYWORD and WRITER_KEYWORD in the mapping file or on the command line. The syntax of these directives is: READER_KEYWORD WRITER_KEYWORD These directives are used most often when the READER_TYPE and WRITER_TYPE are the same. For example, if FME is to read a Design file, alter the geometry of some of the features, then output a new Design file, a mechanism is needed to identify the source and destination portions of the transformation specifications, as well as the source and destination data sets. The following mapping file fragment shows how to configure FME to do such a translation: READER_TYPE IGDS WRITER_TYPE IGDS WRITER_KEYWORD I_OUT # Name the input IGDS file. Note that since no # was specified, the default of IGDS # is used IGDS_DATASET original.dgn # Provide the name for the output IGDS file I_OUT_DATASET newone.dgn # Identify the seed file for the output IGDS file I_OUT_SEED_FILE seed.dgn # Now write a transformation specification. The # source line will be labeled with the # (IGDS), and the destination line # will be labeled with the (I_OUT) IGDS 40 igds_color 3 igds_style %s I_OUT 40 igds_color 4 igds_style %s \ @Generalize(Douglas,10 When a or is specified, the reader or writer module first scans the file for its configuration options using this keyword as a prefix. If any of its required options are not found, then the configuration file is rescanned to search for the option prefixed by the default prefix which is the same as the reader or writer type. In the example above, if the line: I_OUT_SEED_FILE seed.dgn was given as: IGDS_SEED_FILE seed.dgn the end result would be the same. The Design file writer module first scans the mapping file for I_OUT_SEED_FILE, because I_OUT is the . However, it will not find this keyword. It then rescans the file using IGDS as the and finds a value for IGDS_SEED_FILE. Reader Datasets The dataset a reader will extract data from is specified with the _DATASET directive. The syntax for this directive is: _DATASET []+ One or more datasets can be specified, separated by spaces. If a dataset specification contains spaces, it must be enclosed in quotes. When more than one dataset is listed, FME creates a reader for each of the datasets listed, one after the other. However, the features that are read will appear to the rest of FME to come from a single reader. FME's MULTI_READER is used internally to do this merging. Therefore, when multiple datasets are listed, the features will have the MULTI_READER attributes added to them so that their true original source can be determined if necessary. See the Multi-Reader chapter in the FME Readers and Writers manual. If one of the datasets cannot be read, then it is skipped and a warning is logged. If none of the datasets can be read, an error is generated. A can also contain wildcards which FME will expand into a list of files or directories. If the dataset contains any of the metacharacters supported by C-shell glob style syntax, then the dataset is taken as a pattern and expanded to include all files that match. For more information, see Special Characters. Special Characters Special characters supported for a reader dataset are: ? Matches any single character. * Matches any sequence of zero or more characters. ** Matches the current directory and recursively all subdirectories. [chars] Matches any single character in chars. If chars contains a sequence of the form a-b then any character between a and b (inclusive) will match. {ab, cd,e,...} Matches any of the strings ab, cd, e, and so on. Examples: C:\data\*.dgn Expands to all files in the c:\data directory that end with a .dgn extension. C:\data\**\*.dgn Expands to all files in the c:\data directory and any subdirectory below it that ends with a .dgn extension. C:\**\*.dgn Expands to all files on the entire C: drive that end with a .dgn extension. C:\{data,archive}\*.dgn Expands to all files in the c:\data and c:\archive directories that end with a .dgn extension. C:\{data,archive}\92*.dgn Expands to all files in the c:\data and c:\archive directories that start with 92 and end with a .dgn extension. C:\data\92?034.dgn Expands to all files in the c:\data directory that start with 92, have any letter or number next, and end with 034.dgn. C:\data\92[a-z]034.dgn Expands to all files in the c:\data directory that start with 92, have any lowercase letter next, and end with 034.dgn. Reader Pipelines Certain readers employ a set of factories to optimize the features they produce into something easier to work with in a mapping file. In such cases, the factories are stored externally to the mapping file and referenced via the _PIPELINE directive. The syntax for this directive is: _PIPELINE Note: This directive is not normally used by mapping file authors. When this directive is present in a generated mapping file, it should not be altered or adjusted. Defining Output Feature Types in the Writer Most writers in FME require the schema for the output data to be defined before translation begins. This is done through DEF lines, which take this syntax: _DEF . . . For example, for the ESRI Shape format, the feature type defines the file name for the output Shapefile. An ESRI Shapefile named ROADS.shp with attributes LENGTH and NUMLANES can be defined like this: SHAPE_DEF ROADS \ LENGTH char(30) \ NUMLANES char(30) The feature type names, attribute names, and types that are allowed depend on the format. For details, consult each format's writer documentation in the FME Readers and Writers help. Defining Output Feature Types Dynamically (Advanced FME) When writing out a large number of feature types, it can be tedious to create definitions for every feature type. In addition, the output feature types may depend on the input data, and may not be known before translation begins. In this situation, FME can create feature types dynamically, so that DEF lines don't need to be written in a mapping file. Many, but not all, FME writers support dynamic feature type creation. To use this feature, a single DEF line must be given to provide a template for the dynamic feature types. Each feature type that is dynamically created is a copy of the template feature type. The fme_template_feature_type attribute is used to define these dynamic feature types. When a feature is written with a feature type not defined on any DEF line, the writer checks if the fme_template_feature_type attribute is defined on that feature. If it is, the value of that attribute is used as the feature type name for the template that this feature should use. In the example below, we write to the GML2 writer, and use the CreationFactory to simulate reading a large number of feature types. Normally we would have a GML2_DEF section for every feature type from 1 to 100, but in this case we have only a single definition (Template), which is used as a template for every dynamic feature type that will be created. Every one of the 100 layers that will be created in the GML2 file will have a schema identical to that of the Template definition, with a different feature type. READER_TYPE NULL NULL_DATASET null WRITER_TYPE GML2 GML2_DATASET "c:/out.gml" GML2_SCHEMA_MODE CREATE # --------------------------------------------------------------------- # Create 100 features, with the count attribute increasing for each one. FACTORY_DEF * CreationFactory \ 2D_GEOMETRY 0 0 \ CREATE_AT_END no \ NUMBER_TO_CREATE 100 \ OUTPUT FEATURE_TYPE 2DCREATOR_CREATED \ count @Count(counter,0) # --------------------------------------------------------------------- # Set each feature's feature type to the count attribute, and set # the fme_template_feature_type. FACTORY_DEF * TeeFactory \ FACTORY_NAME "2DCREATOR_CREATED -> Template Correlator" \ INPUT FEATURE_TYPE * \ OUTPUT FEATURE_TYPE * \ @Transform(FME_GENERIC,GML2) \ @FeatureType(&count) \ @SupplyAttributes(fme_template_feature_type,Template) # --------------------------------------------------------------------- NULL * GML2 * # --------------------------------------------------------------------- GML2_DEF Template \ gml2_type gml2_point \ attr1 gml2_char(10) \ attr2 gml2_char(10) Limiting the Number of Features Returned To limit the number of features returned, you can optionally use one or both of these keywords: _START_FEATURE_NUM _MAX_FEATURES For example, if you have a dataset with 10 features: ï If neither keyword is specified, all 10 features are read. This is the default behavior. ï If START_FEATURE_NUM is specified and is greater than zero, then all features starting from feature number specified in the keyword are read. For example, ï If START_FEATURE_NUM is 5, all features from 5 to 10 inclusive are read. ï If START_FEATURE_NUM is less than or equal to zero it is always treated as 1. ï If MAX_FEATURES is specified and is greater than zero, then features starting from feature number 1 are read up to a maximum of MAX_FEATURES. If MAX_FEATURES is less than or equal to zero then all features are read. For example, ï If MAX_FEATURES is 5, feature numbers 1 to 5 will be read. ï If MAX_FEATURES is 100, features from 1 to 10 will be read (since the dataset only has 10 features). ï If both START_FEATURE_NUM and MAX_FEATURES are specified and are greater than zero, then the number of features read will depend on their values and number of features in the dataset. Here are some examples, considering the same example 10-feature dataset: START_FEATURE_NUM = 1 MAX_FEATURES = 1 Will read only the first feature. START_FEATURE_NUM = 1 MAX_FEATURES = -1 All features are read. START_FEATURE_NUM = 1 MAX_FEATURES = 10 Reatures from 1 to 10 inclusive are read. START_FEATURE_NUM = 10 MAX_FEATURES = 1 Only the 10th feature is read. START_FEATURE_NUM = 10 MAX_FEATURES = 100 Only the 10th feature is read. START_FEATURE_NUM = 100 MAX_FEATURES = 2 No features are read since the dataset has only 10 features. START_FEATURE_NUM = 6 MAX_FEATURES = 2 Features from 6 to 7 inclusive are read. START_FEATURE_NUM = 6 MAX_FEATURES = 100 Features from 6 to 10 inclusive are read. START_FEATURE_NUM = 1 MAX_FEATURES = 0 All features are read. Limiting Feature Types Returned To limit the type of features returned, you can optionally use this keyword: _FEATURE_TYPES .. where ft1 through n lists the feature type names that the reader will read. Some readers read only those feature types specified, while other readers are less efficient and read everything, but only output the specified feature types. The feature types themselves can be specified with special characters. This allows easy specification of feature types that contain spaces, for example. Sharing Resource Directories Using the environment variable FME_SHARED_RESOURCE_DIR, FME can share resource directories that contain the following subdirectories: Transformers, Formats, Coordinate Systems, and Transformer Categories. Windows ïFME_SHARED_RESOURCE_DIR can point to a shared resource directory. For example: FME_SHARED_RESOURCE_DIR=c:\temp\fred;c:\temp\wilma ïThe Registry entry HKEY_CURRENT_USER/Software/Safe Software Inc./Feature Manipulation Engine/Shared Resource Directory can also point to a shared resource directory. Linux FME_SHARED_RESOURCE_DIR must be set to run workspaces that reference custom transformers, custom formats, and custom coordinate systems. For example: export FME_SHARED_RESOURCE_DIR=~/fred;~/wilma Controlling Arc Strokes The stroking of arcs can be controlled by the FME_STROKE_MAX_DEVIATION directive, which defines the maximum distance between the arc feature and its stroked linear feature. If this directive exists, an arc feature is stroked into a linear feature with the smallest number of points, such that the maximum distance between the arc and the stroked line is less than or equal to the value of this directive. This directive defaults to 0 if not specified. If the value of this directive is smaller than or equal to 0, arc features are stroked into linear features such that the degree per edge is roughly the value of FME_ARC_DEGREES_PER_EDGE, which defaults to 5. (See the @Arc function in the FME Functions and Factories manual for more information on FME_ARC_DEGREES_PER_EDGE.) If the value is greater than one of the primary or secondary radii, arc features are stroked into linear features containing 4 points or less. Geometry Handling FME_GEOMETRY_HANDLING Notes: Before the release of FME 2009, this directive was called FME_USE_RICH_GEOMETRY. It has been renamed to more accurately reflect its function in FME. It is important to understand that, for the most part, the addition of the enhanced geometry model will be transparent to users. You will not have to change the way you use FME or set up a mapping file. The only visible difference should be improved output (if there was room for improvement). Enhanced geometry provides FME with the capability to handle very complex geometry types, such as polygon features whose boundaries include arc features. It also provides the ability to truly hold measures. The FME_GEOMETRY_HANDLING global directive is used by FME to signal whether to produce improved geometric results, where possible. The directive only sets the enhanced geometry mode in readers and factories/functions/transformers that were built before the inception of the enhanced geometry model in FME 2006. Most readers, factories, functions and transformers developed after FME 2006 were built using the enhanced geometry model and hence are not affected by this global directive. In addition, the directive does not guarantee that all features output by a reader/function/factory/transformer will be enhanced; only where it is beneficial. For example, the Shape Reader will only output enhanced geometries if the source shapefile contains measures. Writing of enhanced geometry is handled automatically by all writers. Those writers with native enhanced geometry support will take advantage of the more complex geometries that can be represented, and will maintain, within the constraints of the format, the geometry passed to it. Writers that support only classic geometry will automatically convert any enhanced geometry into classic geometry. This means any geometry (i.e., enhanced or classic) can be sent to any writer. To fully take advantage of writers with enhanced geometry support, features with enhanced geometry must be passed to them. The simplest way to accomplish this is to use a reader developed after FME 2006 and/or set the FME_GEOMETRY_HANDLING directive to YES. The default for this directive is NO. To use the enhanced geometry mode, set the directive to YES. For example: FME_GEOMETRY_HANDLING YES If the directive is set to NO or is missing, the mapping file/workspace/FME Objects application will use classic geometry mode. This means existing workspaces/mapping files/FME Objects applications will produce the same results as they did before the introduction of the enhanced geometry mode. In Workbench and the Viewer, the FME_GEOMETRY_HANDLING option is called Geometry Handling, and valid values for this parameter are Enhanced and Classic. To set this in Workbench, go to the Navigator pane, choose Advanced Settings > Geometry Handling. In the Viewer, select View > Options > Geometry Handling. The default mode is Classic in Workbench and Enhanced in the Viewer. See the documentation of applicable readers, factories, functions and transformers to find more details on how their behavior will change when the geometry mode is set to Enhanced. For more information on the differences between enhanced geometry and classic geometry handling, visit www.fmepedia.com and search Geometry Model. Specifying a Temporary Files Directory FME puts many temporary files in the system temporary directory, which on some Windows versions defaults to C:\temp. If there is not enough room in this directory, you can override this default and send the FME temporary files to a different directory using the FME_TEMP environment variable. (In Windows, look under Control Panel | System | Advanced | Environment Variables). FME searches for a temp directory using the following tests in order: 1.If the user environment variable FME_TEMP exists, use its value as the temporary directory path. 2.If the user environment variable TEMP exists, use its value as the temporary directory path. 3.If a directory called \temp exists on the drive on which FME is running, use that directory. 4.If a directory called \tmp exists on the drive on which FME is running, use that directory. If all of these tests fail to locate an existing temporary directory, FME aborts with an error message indicating that it couldn't open a temporary file. To remedy this situation, either: Create an :\temp or :\tmp directory (where is the drive on which FME is running). OR Define the FME_TEMP or TEMP user environment variable, either system-wide or for the user account on which FME is running. For example, if the temporary files are to be placed into the directory j:\scratch, then the environment variable FME_TEMP would be set to j:\scratch before FME is run. Debugging Mapping Files FME provides several facilities that may be enabled to assist in debugging mapping files. These debugging facilities are useful for finding errors in a custom mapping file that is being developed, or errors in input data. Each debugging option causes diagnostic information to output to the log file. The debug options are enabled by listing the desired debug facilities on one or more FME_DEBUG lines in a mapping file. The syntax of these lines is: FME_DEBUG []* Debugging Options and Related Actions Option Action BADNEWS Valid only for Safe Software developers. When activated, FME logs the location and message of any internal software exception. DonutFactory Activates warning output from the DonutFactory. Points and polygons contained by more than one polygon are logged, as are polygons containing more than one point. This is used to track down problems with extra points or overlapping polygons in the original source data set. DUMP_SCHEMA This option is usually only used on the mapping file generation command line to cause FME to log the schema information it reads from the source dataset(s). This can be used to determine why a mapping file is not being generated in the expected way, usually to troubleshoot problems with generation parameters or the source reader. It is also often used by developers using the plug-in SDK to check if their readSchema implementation is correct. FACTORY Each factory definition line is logged as the factory is activated. This can be used to ensure that the factories intended to operate are correctly created and configured. FACTORY_TRACE Causes each feature to have the attribute fme_factory_trace added to it. This attribute contains an ordered list of all factories the feature passed through. The @Log() function may be used in conjunction with this option, causing this attributeís value to be logged to allow inspection. FME_END_TCL Logs additional diagnostic messages in case the FME_END_TCL directive fails. HTTP_VERBOSE Logs additional output for HTTPFactory, such as a progress counter for the data fetched, and additional header information. HTTP_DEBUG Logs all information provided by HTTP_VERBOSE in HTTPFactory, with additional debugging information, such as details regarding the HTTP request and response connection information. MAPPING_FILE Every line of the mapping file is logged to the log file after all blanks and comments are removed, continuation characters are taken into account, and macros expanded. A histogram of the keywords encountered in the mapping file is also listed and used to view the mapping file as FME actually sees it. The keyword histogram can also be used to discover missing continuation characters if it reports extra incorrect keywords. PolygonDissolveFactory Activates additional statistical output from PolygonDissolveFactory. When a GROUP_BY clause is used with this factory, this debug option causes the number of features input and output from each group to be logged. ReferenceFactory Activates additional statistical output information for ReferenceFactory. A table showing the number of features processed by ReferenceFactory is output. Relate Causes each row read by an @Relate query to be logged to the log file. TRANSFER_PROCESS This is a resource-intensive debugging option that examines the transformation process. It reports any attributes of source features that were ignored or referenced but not defined during the transformation process. Because it is performed on each feature and may produce a great deal of output in the log file, it should be used only when testing mapping files on small input datasets. TRANSFER_SPECS Each transfer specification source and destination line is logged to the log file as it is processed. This is useful when tracking down an unmatched transfer specification pair or to ensure that the transfer specifications are expanded appropriately. UNCORRELATED One feature of each feature type is logged if it was not matched by any source line of a transformation specification. This assists in determining exactly which features were not correlated, so transformation specifications may be written for them. UNGROUPED One feature of each feature type is logged if it was correlated and transformed but had no output destination. For example, if a Shape feature of the type road was created but there was no SHAPE_DEF for the road type, the feature is logged. XSD_VERBOSE Logs detailed XML Schema parsing information. This option can be used to trace the XSD includes and imports, as well as how and where each of the XSD URLs are mapped into local copies. XML Schema parsing may be triggered by the GML readers and writers, the WFS reader, and the XMLSampleGenerator transformer. Identifying Mapping Files For mapping files that may be distributed widely, FME supports a MAPPING_FILE_ID directive that allows the exact version of a mapping file to be echoed in the FME log file. This provides quick confirmation that the correct mapping file version was used when browsing the log file. The syntax is: MAPPING_FILE_ID For example, if a mapping file contains the following line (which should appear as a single, continuous line): MAPPING_FILE_ID TRIM to Shape Release 2.0 Oct 1 ë02 then the string ìTRIM to Shape Release 2.0 Oct 1 ë02î is added to the FME log file each time the mapping file is run. Specifying a Minimum Build Requirement To ensure that a mapping file is not used with an older version of FME that may not support all the facilities used in it, the mapping file may specify the minimum FME build number that can run it. The syntax for the FME_MINIMUM_BUILD directive is: FME_MINIMUM_BUILD For example, if a mapping file contains the line: FME_MINIMUM_BUILD 5600 then no FME with a build earlier than 5600 is permitted to run the mapping file.