SQLExecutor
Executes SQL queries against a database once for each initiating feature that enters the transformer. Schema, the initiating features, and the results of the query may be output.
Typical Uses
- Extract database records as FME features according to a SQL statement
- Execute a database join
- Perform SQL database table operations (create, drop, modify, truncate)
- Create or drop indexes or constraints before or after data loading
How does it work?
The SQLExecutor receives features via the Initiator port, and for every feature it receives, it executes a SQL query against an external database. The query results, as well as the initiating features, are output.
SQL statements are constructed within the transformer, using the appropriate syntax for the database in use. Spatial queries are supported if the database supports spatial predicates.
Schemas may also be produced if supported by the database. One schema feature will be output for every initiator feature.
Examples
In this example, we have a dataset of Fire Halls, which includes point geometry and some attributes. They have a street address, but not a complete postal address.
We will use a SQLExecutor to join a database table (using the Address field as a key), and retrieve the complete postal address.
Additionally, we will perform a second join from database table to database table, and retrieve the Civic Number for each hall.
The FireHalls datasets is connected to the Initiator port on a SQLExecutor. Each FireHalls feature will cause the SQL query to be executed, for that feature.
The first step is to connect to the external database. Select the type, in this case, PostGIS. Then configure (or select, if pre-saved) the database Connection.
Next we create a SQL statement to join both table. Note:
- @value(Address) is providing a value from the Initiator feature - the address for a specific feature, which restricts the results to only matching records for that feature. Since the query is run every time a feature passes through, this repeats for every FireHalls feature.
- The PostalAddress table is being joined to the feature using PostalAddress field.
- The AddressPoints table (which contains the CivicNo attribute) is joined to the PostalAddress table (not the Initiator feature), using a different key - AddressId.
As the attributes will be hidden by default, we choose a set of attributes to expose on our output features.
The features with query results are output via the Results port, and have new attributes added by the SQL statement.
The results for multiple features can be seen in Table View.
Usage Notes
- If the database supports spatial predicates, spatial queries may be used.
- The SchemaScanner may also be useful as an alternative for producing schemas.
- For simple joins of one database table to features, the DatabaseJoiner may be more efficient (if the key field(s) are indexed). The DatabaseJoiner does not require knowledge of SQL.
- The SQLExecutor can only be used on SQL-enabled data sources (in addition to the Initiator features). To use SQL with non-SQL data, consider the InlineQuerier.
- If all the data to be queried already exists in a SQL-capable data source, it is generally most efficient to use the SQLCreator or SQLExecutor, as the queries and filtering of the data is executed directly by the database before it enters the FME environment.
- Features that enter this transformer are not output until the SQL query completes. This behavior ensures that additional SQLExecutor transformers, if present, do not execute their queries until the previous query completes.
Choosing a Feature Joining Method
Many transformers can perform data joining based on matching attributes, expressions and/or geometry. When choosing one for a specific joining task, considerations include the complexity of the join, data format, indexing, conflict handling, and desired results. Some transformers use SQL syntax, and some access external databases directly. They may or may not support list attribute reading and creation.
Generally, choosing the one that is most specific to the task you need to accomplish will provide the optimal performance results. If there is more than one way to do it (which is frequently the case), time spent on performance testing alternate methods may be worthwhile. Performance may vary greatly depending on the existence of key indexes when reading external tables (as opposed to features already in the workspace).
Transformer |
Match By |
Uses SQL Statements |
Can Create List |
Input Type |
Notable |
Description |
---|---|---|---|---|---|---|
FeatureJoiner | Attributes | No | No | Features |
|
Joins features by combining the attributes and/or geometry of features based on common key attribute values. Performs the equivalent of Inner, Left, and Full SQL joins. |
FeatureMerger | Attributes | No | Yes | Features |
|
Merges the attributes and/or geometry of one set of features onto another set of features, based on matching key attribute values and expressions. |
ListBasedFeatureMerger | List Attribute to Single Attribute | No | Yes | Features |
|
Merges the attributes and/or geometry of one set of features onto another set of features, based on matching list attribute values with key attribute values and expressions. |
InlineQuerier | SQL query | Yes | No | Features |
|
Creates a set of SQLite database tables from incoming features, executes SQL queries against them, and outputs the results as features. |
SQLCreator | SQL query | Yes | No | External DB |
|
Generates FME features from the results of a SQL query executed once against a database. One FME feature is created for each row of the results of the SQL query. |
SQLExecutor | SQL query | Yes | No | External DB |
|
Executes SQL queries against a database. One query is issued to the database for each initiating feature that enters the transformer. Both the initiating features and the results of the query may be output as features. |
DatabaseJoiner | Attributes | No | Yes | External DB and Features |
|
Joins attributes from an external table to features already in a workspace, based on a common key or keys. SQL knowledge not required. Non-blocking transformer. |
Matcher | Geometry and/or Attributes | No | Yes | Features |
|
Detects features that are matches of each other. Features are declared to match when they have matching geometry, matching attribute values, or both. A list of attributes which must differ between the features may also be specified. If matching on attributes only (not geometry), using the FeatureMerger or another method will give better performance. |
Configuration
Input Ports
Features that trigger a SQL query to be executed.
Output Ports
If requested in parameters and supported by the database, schema features are output here.
Features that result from the SQL queries.
Input Initiator features with an additional attribute (_matched_records) that contains the number of features generated as the result of SQL query initiated by that feature.
When SQL execution fails, the original Initiator feature is output through this port with an additional attribute called _reader_error containing the last error message.
Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.
Parameters
Format | Select a database format. |
Dataset/Connection | According to chosen format, select a dataset or use/configure a database connection. |
Parameters | Format-specific parameters |
Coord. System |
The coordinate system of the database being connected to (not the Initiator), as well as the coordinate system of the Result output features. Leave as default (Read from source) or use a selection from the Coordinate System Gallery. |
SQL Statement |
Specify the SQL query using the text editor. Multiple SQL commands can be delimited by a character specified using the keyword FME_SQL_DELIMITER, embedded at the very beginning of the SQL statement. The single character immediately following this keyword will be used to split the SQL which will then be sent to the database for execution. (Note: Include a space before the character.) An individual statement may be preceded with a single hyphen, indicating that errors should be ignored. Note: Some database formats support comments with /* ... */ or double hyphens (--). The following example contains two SQL commands where errors from the first command will be ignored: FME_SQL_DELIMITER ; -SELECT * FROM TABLEA; SELECT * FROM TABLEB; |
Attributes to Expose |
Enter the names of attributes to expose on the features created by the query. The attributes will be output in the same sequence as specified in the list. Note: By default, the attributes of the resulting features are hidden. Specify which attributes to expose by entering the attribute names. Click the browse button next to the Attributes to Expose parameter. You can also use a SQL statement to populate the list by pressing “Populate from SQL Query…” and entering a SQL query. The columns from the first matching feature will be used to populate the attributes list. |
Combine Attributes | Result Attributes Only: The result feature attributes consist solely of the query results. Keep Initiator Attributes if Conflict: The result feature attributes are a combination of both the query results and the initiator feature's attributes. If there is a conflict, attribute values are taken from the initiator feature. Keep Result Attributes if Conflict: The result feature attributes are a combination of both the query results and initiator feature's attributes. If there is a conflict, attribute values are taken from the query results. |
Ignore Nulls |
Enabled when Combine Attributes is set to either Keep Initiator Attributes if Conflict or Keep Result Attributes if Conflict. If Yes, where there are attributes with the same name on both Initiator and Result feature, if one has a value of <null> then the other attribute value will be used. If No, the Combine Attributes selection will be strictly observed, including keeping null values. |
Combine Geometry | Result Geometry Only: The geometries of result features consist only of those produced from the SQL query. Initiator Geometry Only: The geometries of result features consist only of those present in the initiator features. Aggregate Initiator and Result Geometry: Result features are populated with geometries from both the initiator features and the SQL query. |
Features to Read |
Select the type of features to read and output:
|
Editing Transformer Parameters
Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.
Defining Values
There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.
Using the Text Editor
The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.
Using the Arithmetic Editor
The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.
Conditional Values
Set values depending on one or more test conditions that either pass or fail.
Parameter Condition Definition Dialog
Content
Expressions and strings can include a number of functions, characters, parameters, and more.
When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.
These functions manipulate and format strings. | |
Special Characters |
A set of control characters is available in the Text Editor. |
Math functions are available in both editors. | |
Date/Time Functions | Date and time functions are available in the Text Editor. |
These operators are available in the Arithmetic Editor. | |
These return primarily feature-specific values. | |
FME and workspace-specific parameters may be used. | |
Creating and Modifying User Parameters | Create your own editable parameters. |
Dialog Options - Tables
Transformers with table-style parameters have additional tools for populating and manipulating values.
Row Reordering
|
Enabled once you have clicked on a row item. Choices include:
|
Cut, Copy, and Paste
|
Enabled once you have clicked on a row item. Choices include:
Cut, copy, and paste may be used within a transformer, or between transformers. |
Filter
|
Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output. |
Import
|
Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers. |
Reset/Refresh
|
Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers. |
Note: Not all tools are available in all transformers.
Reference
Processing Behavior |
Not applicable |
Feature Holding |
Not applicable |
Dependencies | Format-dependent - may require third-party drivers for some formats |
Aliases | |
History |
FME Community
The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.
Search for all results about the SQLExecutor on the FME Community.
Examples may contain information licensed under the Open Government Licence – Vancouver and/or the Open Government Licence – Canada.