FME Form: 2024.2
Typically, features moving through an FME workspace are evaluated individually - but performance gains are possible when multiple features share the same schema and can be processed together, in bulk.
Through bulk mode processing, FME improves performance in many readers, writers, and transformers, as well as other mechanisms such as feature caching and visualization. Sometimes referred to as "feature tables" (as in the FME Objects API), bulk mode optimizes how FME stores and works with features by processing multiple similar features that share the same schema more efficiently - that is, in bulk.
Where is bulk mode supported?
Support for bulk mode is specific to each reader, writer, and transformer. It is possible to read features in bulk, but split them if downstream transformers or writers do not yet support bulk mode.
If supported, FME will always use bulk mode. In most cases, there is no option required to enable bulk mode. In other cases, to get the benefits of bulk mode, you may need to add a new reader or writer to the workspace, or upgrade a transformer.
How do I know when bulk mode is used (or not used)?
It may not always be obvious when bulk mode is supported. One indication is that feature links will likely be emitted in bigger batches at a time instead of scrolling by multiple features.
The log file should indicate when bulk mode is disabled by a warning message when the underlying feature table is split into individual features, indicating a transformer or writer is encountered that does not support bulk mode processing. In these cases, sometimes by using different options or transformers that support bulk mode to achieve the same ends, it may be possible to improve performance.
Can I override bulk mode?
Yes, you can force bulk mode to individual mode with a Decelerator transformer, using Per Feature Delay set to 0 seconds. The feature table will be split into individual features.
What is required for bulk mode to work?
Bulk mode support requires a consistent schema for features:
- Same attribute names and types.
- Static attribute names—that is, an attribute name cannot derive from another attribute.
- Same coordinate system.
In some cases formats or transformers may only read, process, or write in bulk mode if certain options are enabled and the incoming data is suitable for bulk mode processing. For example, the Creator transformer will only process in bulk mode if the number of features requested for creation is greater than or equal to 10000.
How much can bulk mode improve performance?
For bulk mode to work most effectively, the bulk data must be preserved through the workflow. Once created by a bulk mode-enabled reader, the longer the bulk data is preserved by transformers and writers that support bulk mode, the greater the potential performance improvement.
While performance improvements are possible using bulk mode, they are not always achievable for all workspaces and data.
Bulk mode mainly mitigates the cost of creating and destroying multiple features by avoiding having to deal with features individually. So, if creating features is expensive to a workspace, that workspace would likely see an improvement from bulk mode processing. However, if factors other than creating and destroying features consume the majority of resources in a workspace, performance improvements may be less notable.
Generally, the more features are processed and the simpler their schemas (fewer string attributes and fewer attributes in general), the greater the potential improvement. Also, data that contains large geometries such as rasters, point clouds, 3D, or very large vector geometries will benefit less from bulk mode support.
What are the limitations of bulk mode?
For bulk mode to work most efficiently, transformers with multiple output ports—except Feature Holding ("Blocker") transformers—must specify the Per Output Port setting for the Preserve Feature Order parameter. However, in workspaces that rely on feature ordering from joined output ports, this setting produces unexpected results, and is not advised.
How is support for bulk mode expanding?
Adding bulk mode support to the majority of readers, writers, and transformers is ongoing.
Due to expected minimal potential performance benefits or lack of consistent schemas, there may be some formats or transformers that have no plans to be upgraded.