FME Flow: 2024.2

Queue Control

Select Engine Management > Queues, Job Routing Rules, or Engine Assignment Rules.

Queue control is a mechanism for targeting FME Engines to run specific jobs. The reasons for using queue control include:

  • Controlling the priority of job requests.
  • Sending a job to an FME Engine in close proximity to a data source.
  • Reserving FME Engines for scheduled tasks.
  • Reserving Dynamic Engines for specific jobs.
  • Reserving some FME Engines for quick jobs, and others for high-load jobs.
  • Sending a job to an FME Engine that supports a particular format.

How It Works

Queue control is configured through three mechanisms that work together to ensure the right engines run your jobs.

  • Job Routing Rules: The criteria for a job to meet in order to qualify for a specific queue. Criteria can be based on performance metrics of workspaces, which are reported on the Workspaces page. Or, the criteria can be based on other properties of workspaces, such as name or repository.
  • Engine Assignment Rules: The assignment of engines to queues based on properties of engines or queues. Alternatively, engines can be assigned to queues by engine name. Optionally, with Active Periods, engine assignment rules can apply only at specified times.
  • Queues: The FME Flow mechanism that holds jobs and routes them to engines based on job routing and engine assignment rules.

Example: Routing Jobs from a Specified Repository

Consider that you want to run all workspaces in repository Samples on two specific FME Engines.

First, create a queue, named Samples_Queue, that will hold the jobs that are routed to the specified engines. Next, create a job routing rule that routes jobs from the Samples repository to Samples_Queue:

  • Type: Repository,
  • Repositories: Samples
  • Queue: Samples_Queue

Finally, create an engine assignment rule that assigns engines to run jobs routed to Samples_Queue:

  • Type: Name
  • Engines: Select the two FME Engines that you want to assign to queue Samples_Queue.
  • Queues: Samples_Queue

Example: Routing Jobs Based on Performance Metrics

Consider that you want to offload your highest-processing jobs to FME Engines that are installed on a separate server, named "FMEFlow2". On the Workspaces page, inside individual repositories, you can see some useful performance statistics for workspaces that have run jobs to date, or since the statistics were updated. You decide to offload jobs to this server based on a combination of Average % CPU > 80 and Average CPU Time > 1 Hour.

First, you create a separate queue, named queue_FMEFlow2, that will hold the jobs that are routed to the engines on FMEFlow2.

Next, you create a job routing rule that specifies: 1) Your job routing criteria—in this case, %CPU > 80 % AND CPU Time > 1 Hour; and 2) Jobs that meet these criteria are assigned to queue_FMEFlow2. (Note that you can assign a job routing rule to only one queue. A queue, however, can be assigned to multiple job routing rules.)

Finally, you create an engine assignment rule that specifies which engines run jobs that are routed to queue_FMEFlow2. You can specify the engines by name. However, a better option is to specify the engines based on a common system property assigned to all the engines installed on server FMEFlow2. On the Engines tab, you can see that one of the system properties assigned to your FME Engines is the server name on which they are installed. In this case, some of your engines are assigned the property FMEFlow2. This way, if you add more engines to FMEFlow2, they can also run jobs in the queue without explicitly adding them to this rule. (Note that more than one queue can be assigned to an engine assignment rule.)

Things to Consider

When working with queue control, keep in mind the following:

  • Queue priority: You can assign priorities to queues. Depending on timing, jobs that route to higher-priority queues may be submitted before those in lower-priority queues.
  • Job routing priority: A job routing rule can apply to only one queue. However, a queue can be assigned to multiple rules. If a job meets the criteria for more than one rule, the rules that takes precedence are listed in order on the Job Routing Rules tab. For more information, see Prioritizing Job Routing Rules (below).
  • When a job is submitted, it may qualify for more than one queue. However, if that same job is resubmitted, it is assigned the same queue from which it ran previously.
  • Jobs that do not route to any user-defined queues route to the Default queue.
  • For a metric-based job routing rule to take effect, the workspace must have run enough times since statistics are initially collected or reset to generate a statistically significant sample. The required minimum number of jobs to run for a significant sample may vary between 30 to 100, depending on variability of data and the degree of precision required by the rule (for example, minutes versus seconds). The more variable the input data, and the higher degree of precision that is required of this data, the larger the sample size that is required. If a job routing rule does not reference a statistically significant sample of jobs run, the next applicable rule takes effect, by priority, or the job routes to the Default queue.
  • Explicitly defining a queue, such as on the Run Workspace page, a Run a Workspace action, in Schedules, or through Job Directives, effectively bypasses job routing rules.
  • You can view the queues to which your FME Engines are assigned on the Overview or Queues tab of the Engine Management page. Additionally, to assist in balancing engine load, you can set a limit on the number of queues to which engines can be assigned.

Getting Started with Queue Control

To get started with queue control, create your own queues, specify the job routing rules that apply to each queue, and assign engines to run queued jobs.

1. Queues Tab: Create a Queue

  1. On the Queues tab, click Create.
  2. Enter a Name and Description (optional) for the queue.
  3. Assign a Priority. The highest priority is 10 and the lowest priority is 1. Queues with higher priority submit job requests before queues with lower priority, depending on timing.
  4. Click OK.

2. Job Routing Rules Tab: Assign the Queue to a Rule

To assign a queue to an existing rule:

  1. Open the Job Routing Rules tab and click on a rule to open it.
  2. In the Queue field, update the queue to which this rule applies.
  3. Click OK.
Note  You can assign a rule to only one queue.

To assign a queue to a new rule:

  1. Open the Job Routing Rules tab and click Create.
  2. On the Create Job Routing Rule page, specify the following:
    • A Name for the rule.
    • Type:
      • Properties: The queue handles jobs that meet the requirements of specified job performance measures or other properties, including one or more of the following:
        • % CPU: Jobs whose Average % CPU meets the requirements specified in the rule.
        • CPU Time: Jobs whose Average CPU Time meets the requirements specified in the rule.
        • Elapsed Time: Jobs whose Average Elapsed Time meets the requirements specified in the rule.
        • Peak Memory Usage: Jobs whose Average Peak Memory Usage meets the requirements specified in the rule.
        • Repository: Jobs that originate from workspaces in one or more specified Repositories.
        • Workspace: Jobs from one or more specified workspaces. Specify the repository, then the workspace.
        • Source Type: Jobs that originate from one or more specified FME Flow mechanisms.
        • Source Name: Jobs that originate from one or more named FME Flow items, such as the name of a schedule.
        • User Name: Jobs that run under the specified Username. For more information about the Username property, see Jobs (Completed, Queued, or Running).
        • User Role: Jobs that run under any Username belonging to one or more specified Roles. For more information about the Username property, see Jobs (Completed, Queued, or Running).

        Using boolean logic, specify the parameters of the Rule. To add parameters to the rule, click + Rule. To nest parameters in the rule, click + Ruleset.

        For example, to combine Average % CPU > 80 and Average CPU Time > 1 hour, specify % CPU > 80 % AND CPU Time > 1 Hour.

        • Note   For a metric-based job routing rule to take effect, the workspace must have run enough times since statistics are initially collected or reset to generate a statistically significant sample. The required minimum number of jobs to run for a significant sample may vary between 30 to 100, depending on variability of data and the degree of precision required by the rule (for example, minutes versus seconds). The more variable the input data, and the higher degree of precision that is required of this data, the larger the sample size that is required. If a job routing rule does not reference a statistically significant sample of jobs run, the next applicable rule takes effect, by priority, or the job routes to the Default queue.
      • Repository: The queue handles jobs that originate from workspaces in one or more specified Repositories.
        • Note  To combine repository with other criteria, specify Type: Properties, Rule: Repository, specify a repository, then add additional Properties.
    • Queue: Apply this rule to a queue.
    • Enabled: By default, the rule is enabled, which means that subsequent jobs that meet the requirements of the rule are routed to the specified Queue. If disabled, jobs that would otherwise route to this queue based on this rule instead route to the next applicable rule, by priority (see Prioritizing Job Routing Rules, below). If no other rules apply, jobs route to the Default queue.
  3. Click OK.

3. Engine Assignment Rules Tab: Specify How Engines are Assigned to Queues

A set of engine assignment rules determines how engines are assigned to queues. You can assign queues to specific engines by name, or based on the properties of engines and queues.

To add a queue to an existing engine assignment rule:

  1. Open the Engine Assignment Rules tab.
  2. Click the rule to which you want to add the queue.
  3. On the Editing Engine Assignment Rule page, add the queue to the Queues field.
  4. Click OK.

To add a queue to a new rule

  1. Open the Engine Assignment Rules tab, and click Create.
  2. On the Create Engine Assignment Rule page, specify:
    • A Name for the rule.
    • Type:
      • Property: Engines are assigned to queues based on properties of engines and queues.
        • Rule: Using boolean logic, specify the properties that determine how engines are assigned to the specified queue(s).

          • You can specify one or more of the following properties:

          • Dynamic: FME Engines license type is Dynamic.
          • Standard: FME Engines license type is Standard.
          • Custom: In addition to Dynamic and Standard, you can specify other engine properties as they appear on the Engines tab, such as the name or operating system of the host machine.
          • Physical Memory of Engine Host: Amount of RAM on the machine hosting an FME Engine.
          • Processor Count of Engine Host: Calculated as: (number of CPUs) x (number of Cores) x (number of hyperthreads) on the machine hosting an FME Engine.
          • Number of queued jobs: The number of jobs that are currently in a specified queue.
          • Longest queued job wait time: The time of the job that has been waiting the longest in a specified queue.
          • Time for all queued jobs to finish: The time for all jobs in a specified queue to finish based on historical data.
          • CPU for all queued jobs to finish: The CPU time for all jobs in a specified queue to finish based on historical data.
          • Number of Engines: The number of engines assigned to a specified queue. If you are concerned about the engine load of a particular queue based on current performance metrics of the queue's job routing rules, use this property to offload jobs to the queues you specify below, under Queues. For example, if queue MyQueue1 already has 5 engines assigned to it based on current performance metrics, use this property to offload jobs to other engines.

          The Rule drop-down is prepopulated with some properties from which to select. To add more properties to the drop-down, click Create Property. Beside Type, specify a property category, either Custom, Engine, or Queue. If Custom is specified, enter a Property exactly as it appears on the Engines tab. If Queue is specified, select a queue Property, and specify a queue to which the property should apply. If Engine is specified, select an engine host Property.

          Note  For Kubernetes deployments of FME Engines, specify Custom and the name property of the applicable engine deployment. For more information, see Defining FME Engines, Queue Control, and Streams to a Kubernetes Deployment.

          The rule appears beneath the Rule drop-down as you select properties. To add rules, click + Rule. To nest properties in the rule, click + Ruleset.

          For example, to specify any dynamic engine on server WHISTLER (a Custom property), specify Dynamic AND WHISTLER.

      • Name: Engines are assigned to queues by engine name.
        Note  For containerized deployments of FME Flow, such as Kubernetes, we strongly recommend specifying the engine name through the Property setting instead (see above).
        • Engines: Specify the engine names you want to assign to the queue(s).
          • Note  You cannot specify any engines that are assigned to Streams.
    • Queues: Specify one or more queues to route jobs to engines specified in this rule.
    • Enabled: By default, the rule is enabled. If disabled, jobs that would otherwise route to engines based on this rule instead route to the Default queue.
    • Active Periods: Specify the Active Periods during which the engine assignment rule applies. If None, the rule always applies.
  3. Click OK.

Managing Queues and Rules

Viewing and Resetting Queue Statistics

FME Flow keeps track of statistics on completed jobs for each queue. On the Queues tab, select a queue of interest. In the right pane, click Stats. Statistics values are cumulative until explicitly reset. To reset, click Reset Statistics.

Warning  Reset Statistics affects all queues.

Prioritizing Job Routing Rules

A rule can apply to only one queue. However, a queue can be assigned to multiple rules. If a job meets the criteria for more than one rule, the rules that takes precedence are listed in order on the Job Routing Rules tab. To change the priority of a rule, click the Priority icon beside the rule, and select Move Up or Move Down.

Updating the Priorities of Multiple Queues

On the Queues tab, click Edit Priorities. Move the cursor over the Priority field for the queue you want to configure. Use the up/down arrows to change the priority as desired. Click Save.

Limiting the Number of Queues to which Engines can Be Assigned.

You can view the queues to which your FME Engines are assigned on the Overview or Queues tab of the Engine Management page. You may find that some engines bear a heavier queue load than others based on the current configuration of your engine assignment rules. To redistribute this load, you can set a limit on the number of queues to which engines can be assigned.

From the Engine Assignment Rules tab, click Set Limit. On the Queue Limit Per Engine dialog, move the Unlimited Queues slider to the left, specify the Maximum Queues per Engine, and click OK.

Editing Queues

Open the Queues tab and click on the queue you want to edit. In the right pane, click the edit icon (at top right).

Editing Rules

Open the Job Routing Rules, or Engine Assignment Rules tab, depending on what you want to edit. Click on a rule to edit it.

Enabling and Disabling Rules

Open the Job Routing Rules or Engine Assignment Rules tab, depending on what you want to edit. Select one or more rules, click Actions, and select Enable or Disable.

When a job routing rule is disabled, jobs that would otherwise route to a queue based on the rule instead route to the next applicable rule, by priority (see Prioritizing Job Routing Rules, above). If no other rules apply, jobs route to the Default queue.

When an engine assignment rule is disabled, jobs that would otherwise route to engines based on this rule instead route to the Default queue.

Removing Queues

Note  You cannot remove the Default queue.

Open the Queues tab. Select the queue you want to remove. Remove. In the right pane, click the remove icon (at top right).

Removing Rules

Open the Job Routing Rules,or Engine Assignment Rules tab, depending on what you want to remove. Select one or more or rules, click Actions, and select Remove.