PythonCaller

Executes a user-supplied Python script to manipulate features.

Typical Uses

Tasks where a transformer is not available
Using external modules for processing
Performing complex manipulations on list attributes

How does it work?

The PythonCaller executes a Python script to manipulate features.

When a specialized task is required, such as custom statistical analysis of an attribute, but FME Workbench does not provide a transformer suited to the task, a Python script can perform specialized and complex operations on a feature's geometry, attributes, and coordinate system.

Access is provided via the FME Objects Python API.

Note Python is a programming language external to FME. For documentation on creating Python scripts, visit The Python Foundation.

Using Python to perform arbitrary operations on features is a powerful aspect of FME Workbench. However, the logic introduced into a workspace is less visible and can therefore be more difficult to maintain than logic built using FME Workbench’s built-in transformers. It is recommended that other transformers be used when possible instead of Python scripts.

Class Interface

The PythonCaller can interface with a class defined in a Python script.

The supplied script contains a class called FeatureProcessor, which inherits from fme.BaseTransformer. This is a base class that represents the interface expected by the FME infrastructure. See the FME Python module documentation for more information.

The calling sequence on the methods defined in a class depends on which mode the PythonCaller is in. There are two modes - Standard and Group Processing.

Standard Mode

This is the operation mode when no attributes are set in the Group By parameter. Most uses of the PythonCaller will use standard mode processing. In this mode, the PythonCaller will have the following calling sequence on a class:

__init__() - Called once, whether or not any features are processed.
input() - Called for each FMEFeature that comes into the input port.
close() - Called once, after all features are processed (when no more FMEFeatures remain). If no features are processed, the close() method will still be called.

Features that need to continue through the workspace for further processing must be explicitly written out using the pyoutput() method. When using the pyoutput() method, it is recommended to set the output_tag argument to PYOUTPUT.

As of FME 2023.1, bulk mode is enabled by default for the PythonCaller. This allows for significant performance gains when processing large numbers of features. In bulk mode, FME will pass features to the `input()` method which come from a feature table object. The following conditions must be met:

features passed into the input() method cannot be copied or cached for later use
features cannot be read or modified after being passed to the pyoutput() method
Only features passed into the transformer can be output from the transformer. Newly instantiated FMEFeature() objects should not be output from the transformer. Not every feature passed into the transformer needs to be output from the transformer.
Group processing must not be enabled

Violations will cause undefined behavior. If PythonCaller usage does not meet these requirements, bulk mode should be disabled by altering the has_support_for() method to return False when the support_type argument is equal to fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM.

When the PythonCaller implementation processes incoming FMEFeatures one at a time, the pyoutput() method is to be called once per incoming FMEFeature in the input() method.

When the PythonCaller implementation requires knowledge of multiple FMEFeatures to process features, the incoming FMEFeatures can be stored in a list, then processed and written out through pyoutput() in the close() method.

The example below calculates the total area of all the features processed and then outputs all the features with a new attribute containing the total area. Note that since features are cached for later use, bulk mode should not be enabled.

Group Processing Mode

This is the feature processing mode used when one or more attributes are set in the Group By parameter. When using this mode, bulk mode must be disabled by altering the has_support_for() method to return False when the support_type argument is equal to fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM.

In this mode, the PythonCaller will have the following calling sequence on a class:

__init__() - Called once, whether or not any features are processed.
input() - Called for each FMEFeature in a group.
process_group() - Called after all FMEFeatures in a group have been sent to input(). After this is called and executed, PythonCaller will send the next group of FMEFeatures to input() and then call process_group() again. This is repeated until all groups have been exhausted.
close() - Called once, after all rounds of input() and process_group() have been called to process all incoming features (when no more FMEFeatures remain). If no features are processed, the close() method will still be called.

In this mode, the class interface will be working on groups of FMEFeatures. The incoming FMEFeatures must be stored in a list class member variable, then processed and written out through pyoutput() in the process_group() method. After processing is complete in process_group(), all class member variables should be cleared for the next round of group by handling. In the next round, FMEFeatures of the next group are passed through input() calls followed again by process_group(). This is repeated until all the groups are exhausted.

The example below calculates the total area of all features grouped by the _shape attribute and then outputs all the features with a new attribute containing the total area for each group:

Feature Rejection

Regardless of processing mode, if there are input features that cause operations to fail, they can be sent to the <Rejected> port by using the reject_feature() method defined in the FeatureProcessor class.

Script Editing

A PythonCaller transformer can call scripts that are stored in the transformer itself or scripts that are stored globally for the entire workspace:

To store a Python script with a specific PythonCaller transformer, use the Python Script parameter in the transformer.
To store a Python script globally, click the Advanced Workspace Parameter in the Navigator, and double-click Startup Python Script. Storing scripts globally has the advantage of keeping the Python logic centralized, which makes editing and maintenance easier. This is useful when there are multiple PythonCaller transformers throughout the workspace that use the same script. For more information, see Startup and Shutdown Python Scripts in the FME Workbench help.

FME can access .py modules that are stored on the file system, including modules in external Python libraries. Use the Python "import" command to load these modules. FME will search both the standard Python module locations and the workspace location to find the module to be imported.

Configuration

Input Ports

Output Ports

Parameters

Group Processing (Optional)

Group By

If the Group By parameter is set to a set of attributes, one feature per group will be produced.

Complete Groups

Select the point in processing at which groups are processed:

When All Features Received	This is the default behavior. Processing will only occur in this transformer once all input is present.
When Group Changes (Advanced)	This transformer will process input groups in order. Changes of the value of the Group By parameter on the input stream will trigger processing on the currently accumulating group. This may improve overall speed (particularly with multiple, equally-sized groups), but could cause undesired behavior if input groups are not truly ordered.

Output Attributes

Attributes to Expose

Exposes attributes so they can be used elsewhere in the workspace.

Attribute names may be entered directly or provided in the Enter Values for Attributes to Expose dialog accessed via the ellipsis button, where data type may also be specified.

For more information on exposed and unexposed attributes, see Understanding Feature Types and Attributes.

Used to expose any attributes that are created by the Python script being executed so they can be used by other transformers.

Attributes to Hide

Hides any attributes that may be removed by the Python script being executed. Other transformers will not be able to use these attributes.

Lists to Hide

Hides any lists that may be removed by the Python script being executed. Other transformers will not be able to use these lists.

Note that if you select to hide a list, your selection will include any list attributes or nested lists. For example, if you select to hide a list called

list{}

then list{}.attr or list{}.sublist{}

will also be hidden.

Editing Transformer Parameters

Transformer parameters can be set by directly entering values, using expressions, or referencing other elements in the workspace such as attribute values or user parameters. Various editors and context menus are available to assist. To see what is available, click beside the applicable parameter.

How to Set Parameter Values

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters.

Using the Text Editor

The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.

Text Editor

Using the Arithmetic Editor

The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.

Arithmetic Editor

Conditional Values

Set values depending on one or more test conditions that either pass or fail.

Parameter Condition Definition Dialog

Content

Expressions and strings can include a number of functions, characters, parameters, and more.

When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.

Content Types

String Functions	These functions manipulate and format strings.
Special Characters	A set of control characters is available in the Text Editor.
Math Functions	Math functions are available in both editors.
Date/Time Functions	Date and time functions are available in the Text Editor.
Math Operators	These operators are available in the Arithmetic Editor.
FME Feature Functions	These return primarily feature-specific values.
FME Parameters	FME and workspace-specific parameters may be used.
Creating and Modifying User Parameters	Create your own editable parameters.

Dialog Options - Tables

Table Tools

Transformers with table-style parameters have additional tools for populating and manipulating values.

Row Reordering

Enabled once you have clicked on a row item. Choices include:

Add a row
Remove a row
Move current row up one
Move current row down one
Move current row to top
Move current row to bottom

Cut, Copy, and Paste

Enabled once you have clicked on a row item. Choices include:

Cut a row - delete and copy to clipboard
Copy a row to the clipboard
Paste a row from the clipboard

Cut, copy, and paste may be used within a transformer, or between transformers.

Filter

Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output.

Import

Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers.

Reset/Refresh

Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers.

Note: Not all tools are available in all transformers.

For more information, see Transformer Parameter Menu Options.

Reference

Processing Behavior	Feature-Based or Group-Based, conditional on Python script
Feature Holding	Conditional on Python script
Dependencies	Specifying a Python Interpreter An FME installation includes a Python version 3.13 interpreter. The FME Objects Python API supports Python 3.9 - 3.13. The Python interpreter used by FME to execute Python scripts is controlled by the Python Compatibility workspace parameter . FME will load the latest available Python interpreter matching that parameter. For more information, see FME Workbench help. Installing Python Packages If you would like to install a third-party package for use by Python in FME, see Installing Python Packages to FME Form in FME Workbench help.
Aliases
History

FME Community

The FME Community has a wealth of FME knowledge with over 20,000 active members worldwide. Get help with FME, share knowledge, and connect with users globally.

Search for all results about the PythonCaller on the FME Community.

Examples may contain information licensed under the Open Government Licence – Vancouver, Open Government Licence - British Columbia, and/or Open Government Licence – Canada.

Class to Process Features	The name of the Python Class within the script that PythonCaller will use to begin execution. For the above examples, set this parameter to FeatureProcessor.
Python Script	The Python script to be executed. When the Python script is stored as the Startup Python Script for the workspace, leave this parameter blank.