Streams

Use the Streams page to manage jobs you want to run continuously. When a job runs as a stream, it restarts automatically, regardless of whether the job completed successfully, failed, or the server crashed or was shut down.

Workspaces powered by streams usually have an event queue or message broker as a source. (The workspace must, in fact, contain a transformer that can run in Mode: Stream, such as AzureQueueStorageConnector or KafkaConnector.) The stream allows you to integrate and analyze data in real-time as it is produced. With stream mode, there is no overhead for starting and stopping jobs, which means a very high volume of data can be processed with low latency.

Streams provide the following additional advantages over running jobs continuously through other interfaces:

  • When you create a stream, the FME Engines you assign to run its jobs are dedicated to that stream.
  • You can scale processing throughput by assigning more than one FME Engine to a stream.

    • Note: If the FME Engines you specify for the stream belong to any queues, they are removed from those queues so that they are dedicated to the stream.

  • You can configure a stream and then pause it.
  • The Streams page shows you all the jobs you run continuously, so you can manage them in one convenient place.

Getting Started with Streams

To create a stream, select Streams > Build Stream. Or, select Streams > Manage Streams and click New.

  1. Complete the following fields:
    • Name: Provide a name for the stream.
    • Description: (Optional) Provide a meaningful description of the stream, such as the workspace it runs and why you want it to run continuously. Use the Markdown tools as desired.
    • Repository: Specify the repository that contains the workspace you want to run continuously.
    • Workspace: Specify the workspace you want to run continuously.

      • Note: The workspace must contain a transformer that can run in Mode: Stream, such as AzureQueueStorageConnector or KafkaConnector.

    • Published Parameters: If the workspace has any published parameters, specify them here.
  2. Click OK.
  3. A page opens that summarizes the properties of the stream you created. The stream is ready to run as soon as you assign FME Engines to run the jobs. If you do not want the stream to run immediately, slick Stop.
  4. To assign FME Engines to run the jobs associated with the stream, click Assign under Assigned Engines. When you assign engines to a stream, keep in mind the following:
    • You can only assign engines that are not already dedicated to other streams.
    • If the FME Engines you specify for the stream belong to any queues, they are removed from those queues so that they are dedicated to the stream.
  5. Once you assign engines to a stream, it begins to run, unless you stopped it previously. When you are ready to run the stream, click Start; or from the Manage Streams page, select the stream and click Actions > Start.

Managing Streams

Select Streams > Manage Streams.

To view the jobs that have been run by a stream

Click the stream to open it, and scroll to Jobs.

To view all the engines that are assigned to streams

You can view all the engines that are assigned to streams on the Engines page.

To change the engines assigned to a stream

Click the stream to open it. Under Assigned Engines, click Assign to assign engines. To unassign engines, select the engines and click Unassign.

To start, stop, or remove streams

Select the streams, click Actions, and select Start, Stop, or Remove, respectively.