Creating bulk mode features

Note

This functionality is new to FME 2025.

By default, transformers are unable to create new bulk mode features. CREATE_BULK_FEATURES is a factory clause that allows FME to accumulate output features and assemble them into a bulk mode feature, also referred to as a Feature Table. This can greatly improve performance when the transformer is producing large datasets in FME pipelines.

Tip

For more information on CREATE_BULK_FEATURES, see the PythonFactory section of the FME Factory and Function Documentation.

Tip

For an introduction on what bulk mode features are, see How FME Improves With Bulk Mode.

When to opt in

Use CREATE_BULK_FEATURES when your transformer:

• Reliably produces ≥ FME_BULK_MODE_THRESHOLD output features per input
• Produces output with a consistent coordinate system across features

Do not use CREATE_BULK_FEATURES when your transformer:

• Is a streaming transformer, which continuously waits for incoming features and never reaches a natural end. Without a pipeline flush, output features may be held until the upper threshold for bulk mode features (~100k) is reached, introducing significant latency for downstream transformers
• Produces output with highly variable coordinate systems — a coordinate system change causes the Feature Table to split

Semi-blocking behavior

CREATE_BULK_FEATURES YES causes the factory to hold output features until a flush condition is met.

Flush conditions include:

1. Accumulated feature count reaches the upper threshold (~100k)
2. Output schema changes incompatibly (primarily a coordinate system change)
3. The output port changes. This can be overridden per-factory with PRESERVE_FEATURE_ORDER PER_OUTPUT_PORT. For more details on the PRESERVE_FEATURE_ORDER clause, refer to the PythonFactory section in the FME Factory and Function Documentation
4. All upstream transformers finish and the pipeline flushes

If fewer than FME_BULK_MODE_THRESHOLD features have accumulated at flush time, they are output as individual features instead.

Warning

Even a non-streaming transformer with CREATE_BULK_FEATURES YES will block if a streaming transformer sits upstream and prevents the flush from firing. All transformers using CREATE_BULK_FEATURES should expose a Feature Output parameter (see below) as an escape hatch.

Adding support for bulk feature creation

Here are the recommended steps to adding this functionality.

Adding a Feature Output parameter

All transformers using CREATE_BULK_FEATURES should first expose a Feature Output parameter so users can revert to individual feature output when needed. To follow the convention used by existing FME transformers, add it as a choice parameter under an Advanced collapsible group in the Transformer Designer:

Setting	Value
Attribute Name	___XF_FEATURE_OUTPUT
Display Name	Feature Output
Type	Choice
Default Value	NO
Group	Advanced

Configure the lookup choices as follows:

Display	Value
Individual Mode	NO
Bulk Mode	YES

Defaulting to Individual Mode ensures the transformer is non-blocking by default, and users must explicitly opt into bulk mode.

Note

Adding a new parameter will expose it as an attribute in the output schema. If following the FME Package transformer template used in the Hello World guide, the ___XF_ prefix ensures that this attribute is stripped by the existing @RemoveAttributes call and no additional cleanup is needed.

Adding the CREATE_BULK_FEATURES clause

In the transformer's execution instructions, add the CREATE_BULK_FEATURES clause to the PythonFactory, setting it to $(___XF_FEATURE_OUTPUT) to bind it to the Feature Output parameter:

Execution Instructions

FACTORY_DEF {*} PythonFactory
    FACTORY_NAME { $(XFORMER_NAME) }
    $(INPUT_LINES)
    SYMBOL_NAME { my_library.my_module.TransformerImpl }
    PY_OUTPUT_TAGS { Output <Rejected> }
    CREATE_BULK_FEATURES { $(___XF_FEATURE_OUTPUT) }
    OUTPUT { Output FEATURE_TYPE $(OUTPUT_Output_FTYPE)
        @RemoveAttributes(fme_regexp_match,^___XF_.*$)
        $(OUTPUT_Output_FUNCS) }
    OUTPUT { <Rejected> FEATURE_TYPE $(OUTPUT_<Rejected>_FTYPE)
        @RemoveAttributes(fme_regexp_match,^___XF_.*$)
        $(OUTPUT_<Rejected>_FUNCS) }

With both changes in place, users selecting Bulk Mode will cause the transformer to accumulate output features into a Feature Table, and can switch back to Individual Mode at any time if bulk mode causes unexpected blocking in their workspace.