Use the Streams page to manage jobs you want to run continuously. When a job runs as a stream, it restarts automatically, regardless of whether the job completed successfully, failed, or the server crashed or was shut down.
Workspaces powered by streams usually have an event queue or message broker as a source. Messages are handled by transformers such as AzureQueueStorageConnector or KafkaConnector, that can run in Mode: Stream. The stream allows you to integrate and analyze data in real-time as it is produced. With stream mode, there is no overhead for starting and stopping jobs, which means a very high volume of data can be processed with low latency.
Note: There is no restriction on the type of workspace you can run as a stream. The workspace is not required to contain a transformer that runs in Mode: Stream. However, jobs that contain such workspaces may not behave as expected. Excessive job logging can cause storage issues since the job is resubmitted every time it finishes.
Streams provide the following additional advantages over running jobs continuously through other interfaces:
- When you create a stream, the FME Engines you assign to run its jobs are dedicated to that stream.
- You can scale processing throughput by running jobs from the same workspace simultaneously by multiple engines. The number of jobs a stream runs is equal to the number of engines assigned to the stream.
- You can configure a stream and then pause it.
- The Streams page shows you all the jobs you run continuously, so you can manage them in one convenient place.
Note: If the FME Engines you specify for the stream belong to any queues, they are removed from those queues so that they are dedicated to the stream.
Getting Started with Streams
To create a stream, select Streams > Build Stream. Or, select Streams > Manage Streams and click New.
- Complete the following fields:
- Name: Provide a name for the stream.
- Description: (Optional) Provide a meaningful description of the stream, such as the workspace it runs and why you want it to run continuously. Use the Markdown tools as desired.
- Repository: Specify the repository that contains the workspace you want to run continuously.
- Workspace: Specify the workspace you want to run continuously.
- Published Parameters: If the workspace has any published parameters, specify them here.
Note: The workspace must contain a transformer that can run in Mode: Stream, such as AzureQueueStorageConnector or KafkaConnector.
- Assigning multiple engines means you can scale processing throughput by running the same job simultaneously. The number of jobs a stream runs is equal to the number of engines assigned to the stream.
- You can only assign engines that are not already dedicated to other streams.
- If the FME Engines you specify for the stream belong to any queues, they are removed from those queues so that they are dedicated to the stream.
- If an engine disconnects from FME Server (for example, a machine that hosts a separate FME Engine crashes), any running job reverts to a "queued" state, and is returned to the engine when it becomes available.
- Type: How you want to assign FME Engines to the stream:
- Name: Engines are assigned to the stream by engine name
- Property: Engines are assigned to the stream based on their properties.
- Engines: If Type is Name, select one or more engines by name to assign to the stream.
- Rule: If Type is Property, use boolean logic to specify the properties that determine how engines are assigned to the stream.
- Dynamic: FME Engines license type is Dynamic (CPU Usage).
- Standard: FME Engines license type is Standard.
- Physical Memory of Engine Host: Amount of RAM on the machine hosting an FME Engine.
- Processor Count of Engine Host: Calculated as: (number of CPUs) x (number of Cores) x (number of hyperthreads) on the machine hosting an FME Engine.
The following properties are available to select without manual entry:
To add one of these properties, click + Rule and select it.
Additionally, you can add other engine properties as they appear on the Engines tab, such as the name or operating system of the host machine. To add other properties to the Rule drop-down, click + Create Property. Beside Type, specify Custom, and enter a Property exactly as it appears on the Engines tab.
As soon as you specify a property, the rule appears beneath the Rule drop-down. To add more properties, click + Rule. To nest properties in the rule, click + Ruleset.
For example, to specify any dynamic engine on server WHISTLER (a Custom property), specify Dynamic AND WHISTLER.
When finished, click OK.
Select Streams > Manage Streams.
To start, stop, or remove streams
Select the streams, click Actions, and select Start, Stop, or Remove, respectively.
When a stream is stopped, the associated workspace does not submit any jobs to the stream. Any jobs that are queued or running are canceled.
To view the jobs that are running or in queue through a stream
Click the stream to open it, and scroll to Jobs. Click on a job to open it in the Jobs page.
Tip: Use the Status drop-down to filter the jobs that appear, as follows:
All: Queued and Running jobs.
Running: Jobs that are currently being run by an engine.
Queued: Jobs that are submitted but not yet picked up by an engine to run.
To view all the engines that are assigned to streams
You can view all the engines that are assigned to streams on the Engines page.
To change the engines assigned to a stream
Click the stream to open it. Under Assigned Engines, click Manage to assign engines. Newly-assigned engines run jobs shortly after assignment.