FME Flow: 2025.1 BETA

Data Virtualization

With just basic knowledge of RESTful coding conventions, you can create data virtualization APIs that harness the power of FME workspaces to access and transform data from HTTP endpoints, while conforming to the OpenAPI standard.

The Data Virtualization page lists the APIs that are created. You can:

  • Filter the list by title, namespace, and/or owner.
  • Test an API and View its Documentation: Select its corresponding checkbox, click Actions, and select View Documentation.
  • Download an OpenAPI .json specification file of an API: Select its corresponding checkbox, click Actions, and select Export OpenAPI Specification.
  • View an aggregated log file of Data Virtualization APIs: Select the checkboxes of one or more APIs and click View Log File (requires Access and List permission to the Logs folder in Resources).
  • Remove one or more APIs: Select its corresponding checkbox, click Actions, and select Remove.

How It Works

The easy-to-use interface gives you all the tools you need to define endpoints and configure them with "starter" workspaces. Once your endpoints and workspaces are configured, download the workspaces to FME Workbench and configure your data access and transformation requirements. Your API is complete when you republish your finished workspaces to FME Flow Data Virtualization.

Getting Started

Follow these steps to create a Data Virtualization API:

  1. Create the API: Give the API a name and namespace. Configure initial security, caching, and provide other metadata. See Creating and Managing Data Virtualization APIs, below.
  2. Configure API functionality: Define endpoints and configure with "starter" workspaces. If desired, define reusable data property schemas for endpoints. Adjust security and metadata as desired. See Configuring API Functionality, below.
  3. Download workspaces to FME Workbench, configure data access and transformation requirements, and republish to FME Flow Data Virtualization. These steps are handled in FME Workbench. For more information, see the following topics in the FME Workbench help:

Creating and Managing Data Virtualization APIs

To create an API directly in FME Flow, click Create. To import an API that you can tailor for FME Flow Data Virtualization, click Import. (If there are no previously-created APIs to display, you can also click Create API or Import OpenAPI under Get Started with APIs.)

On the Create API page or Import dialog, complete the following fields. When finished, click Create on the Create API page, or Upload on the Import dialog.

  • API Title: (for APIs created directly in FME Flow only) Give the API a title.
  • Namespace: Provide a namespace that is unique to the API. The namespace is used to route requests to this specific API.
    • Note  Values status and result are reserved and cannot be specified.
  • Choose a .json file: (for APIs imported to FME Flow only) Drop or select the .json file that comprises the API you want to import.

For APIs created directly in FME Flow, the following additional fields may be completed. For all APIs created in FME Flow, these properties may also be configured later. See "To Edit a Data Virtualization API", below.

  • Version: This field autopopulates to track versions of the API.
  • Summary: (Optional) Provide a summary description of the API.
  • Description: (Optional) Provide a detailed description of the API.
  • Job Queue: (Optional) The queue in which to run the workspaces associated with the API.
  • Job Expiry Time: Maximum time to run the workspaces associated with the API, including time in queue. The job is cancelled if it exceeds the specified duration.
  • Name: (Optional) Assign a contact person or organization for the API.
  • URL: (Optional) A URL address that points to the contact Name.
  • Email: (Optional) Email address for the contact Name.
  • Access Level: (Optional)
    • Authenticated: The endpoints of this API are accessible only to the Roles and Users specified under Allowed Roles and Allowed Users (below).
    • Unauthenticated: The endpoints of this API do not require authentication to use.
      • Note  You can update this setting later on the Security tab of the API after it is created. Additionally, when you create individual endpoints, you can configure security that overrides access at the API level.
  • Allowed Roles: (Optional) If Access Level is Authenticated, specify the roles to grant access to this API's endpoints.
  • Allowed Users: (Optional) If Access Level is Authenticated, specify the users to grant access to this API's endpoints.
  • Cache Responses: Specify default caching requirements for workspaces that respond to GET operations of the API's endpoints.
    • Note  You can override this setting when creating or editing individual endpoints.
    • No: Endpoint responses are not cached.
    • Yes: Endpoint responses are cached according to the specified Time to Live (TTL) value in Seconds, Minutes, or Hours.
      • Note  Caches are accessible on the Resources page, under System > temp > cache.
  • Return Asynchronous Response:

    • If Yes, each endpoint of the API returns a status and URL to view results immediately after execution completes. These response results are kept according to the Result Expiry Time, specified in Minutes, Hours, or Days. To view a schema of the response format, click Preview Response.

    If No (default), asynchronous response results are not available for the individual endpoints of the API.

    Note  This setting can be overridden for individual endpoints.

Click on an API to open it.

To edit the API Title or Version, click the edit (pencil) icon beside the API Title.

To edit other properties, open the corresponding tab (Endpoints, Schemas, Security, API Details), and click the edit (pencil) icon corresponding to the section you want to edit. On the Security tab, you can also edit security for specific endpoints. Click the ellipsis (...) under ACTIONS corresponding to the endpoint you want to edit, and select Edit Security.

You may want to configure any URLs that will interact with your API for CORS. For more information, see Cross-Origin Resource Sharing.

You can download the OpenAPI .json specification file of the API. From the Data Virtualization home page, select its corresponding checkbox, click Actions, and select Export OpenAPI Specification.

When you create an API, documentation for its endpoints and schemas is also created. To test an endpoint, expand it and click Try it out.

To view API documentation, do any of the following:

  • From the Data Virtualization home page, select the checkbox for the API, click Actions, and select View Documentation.
  • From the page of the API you opened, click Actions and select View Documentation.
  • On the API Details tab of the API, click View Documentation (under Description).

If caching is enabled for the API overall, or for any of its individual endpoints, you can purge all caches, as follows:

  1. Click on the API to open it.
  2. Open the API Details tab.
  3. Under Caching, click Purge Cache.
  4. On the Purge Cache dialog, click Purge.
Note  Caches are also purged when any of the following occurs:
  • FME Flow is restarted.
  • A workspace is republished to FME Flow Data Virtualization. Any cache associated with the previous version is cleared.
  • An update to the Time to Live setting that renders the cache older than the updated setting allows.

Configuring API Functionality

Most of the functionality for your API is configured in the Endpoints and Workspaces tabs. The Schemas, Security, and API Details tabs provide additional ways to control and configure your API.

  • Endpoints: Define the HTTP methods for receiving and processing requests for data.
  • Workspaces: Generate and manage workspaces that handle endpoint responses.
  • Schemas: Define reusable data property configurations for endpoints.
  • Security: Edit who can access the API and its individual endpoints.
  • API Details: Edit API metadata and caching.

Creating and Managing Schemas

Use the schemas tab to create schemas for defining data properties of HTTP requests in your endpoints, including the name of the data point, data type, and if the data is required. Schemas make it easy to reuse these data points across endpoints.

To Create a Schema

  1. On the Schemas tab for your API, click Create.
  2. On the Create Schema pane, provide a Name and Description (optional) for the schema.
  3. To add a data property to the schema, click Create beside Properties.
  4. Specify a Name, Type of data (optional), and if the data is Required. The specified Type (String, Number, Integer, or Boolean) functions according to its respective OpenAPI specification. If not specified, the schema accepts any type.
  5. Click Create.

To Edit a Schema

On the Schemas tab for your API, click the schema name you want to edit.

To remove a property, click the remove icon (trash can) beside the property.

To Remove a Schema

On the Schemas tab for your API, click the ellipsis (...) beside the schema you want to remove, and select Delete.

Creating and Managing Endpoints

HTTP endpoints specify how your API receives and processes requests for data.

To Create an Endpoint

  1. On the Endpoints tab for your API, click Create.
  2. On the Details tab under CREATE ENDPOINT, provide the following:
    • Path: The URL for the endpoint.
    • Operation: Specify the operation of the endpoint: GET, POST, PUT, or DELETE.
    • Summary (optional): Provide a useful summary of the endpoint's operation, if desired.
    • Description (optional): Provide a more detailed summary of the endpoint's operation, if desired.
    • Endpoint Tags (optional): Assign one or more tags to the endpoint, if desired. Tags are used to organize endpoints into meaningful groups on the Endpoints tab, based on your business rules. To create a tag before assigning it, click Create Tag.
    • Endpoint Security: If you want access to the endpoint to be restricted under the same terms as those specified on the Security tab of the API, enable Inherit API Setting. To configure access separately for this endpoint, disable this setting, and specify:
      • Access Level: (Optional)
        • Authenticated: The endpoints of this API are accessible only to the Roles and Users specified under Allowed Roles and Allowed Users (below).
        • Unauthenticated: The endpoints of this API do not require authentication to use.
      • Allowed Roles: (Optional) If Access Level is Authenticated, specify the roles to grant access to this API's endpoints.
      • Allowed Users: (Optional) If Access Level is Authenticated, specify the users to grant access to this API's endpoints.
        • Note  On the Security tab of the API, you can always update security for specific endpoints after they are created.
  3. If Operation is POST or PUT, click the Request Body tab under CREATE ENDPOINT, and provide the following:
    • Required: Specify if the request body is required to perform POST or PUT operation.
    • Description (optional): Provide a useful description of the request body, if desired.
    • Content Type:
      • application/json: The request body is in the form of JSON key-value pairs.
    • Input Type:
      • Use Schema: The data properties of the request body are defined by the specified schema of the API.
      • Create Properties: The data properties of the request body are defined manually. To create a property, click Create beside Properties, and specify a NAME, TYPE of data (optional), and if the data is REQUIRED. The specified TYPE (String, Number, Integer, or Boolean) functions according to its respective OpenAPI specification. If not specified, the property accepts any type.
  4. Click the Response tab under CREATE ENDPOINT, and specify the following:
    • Response Type:
      • Workspace: Use an FMEworkspace to handle the endpoint response. If a workspace is not yet generated for this endpoint, click Create to save the current endpoint configuration and proceed to Generating and Managing Workspaces for Handling Endpoint Responses. When finished, return here and complete this step (see To Edit an Endpoint).

        Job Settings

        • Inherit API Job Queue: If you want the endpoint to the run the workspace in a different queue from the one that was specified for the API, click to move the widget to the left and specify a Job Queue in which to run the endpoint.
        • Inherit API Job Expiry Time: If you want the endpoint to have a different Job Expiry Time from the one that was specified for the API, click to move the widget to the left and specify a Job Expiry Time specific to this endpoint in Seconds, Minutes, or Hours.

        Status Code

        If the workspace is configured to handle specific status codes, click Add Status Code and specify:

        • HTTP Status Code: The HTTP Status code to handle. Select from the drop-down of status codes from the Open API specification. Alternatively, enter a custom code that conforms to the OpenAPI specification and is specific to your application.
        • Description (optional): Provide a useful description of the response, if desired.
        • Content Type:
          • application/json: The response body is in the form of JSON key-value pairs.
              • Input Type:
                • Use Schema: The data properties of the response body are defined by the specified schema of the API.
                • Create Properties: The data properties of the response body are defined manually. To create a property, click Create beside Properties, and specify a NAME, TYPE of data (optional), and if the data is REQUIRED. The specified TYPE (String, Number, Integer, or Boolean) functions according to its respective OpenAPI specification. If not specified, the property accepts any type.
          • String: The response body is in the form of a string.

        Asynchronous Processing

        If you want the response of the endpoint to differ from the Return Asynchronous Response setting specified for the API overall, click to move the Inherit API Setting widget to the left and specify:

        • Yes: The endpoint returns a status and URL to view results immediately after execution completes. Response results are kept according to the Result Expiry Time, specified in Minutes, Hours, or Days. To view a schema of the response format, click Preview Response.
        • No: Asynchronous response results are not available for this endpoint.
      • Manual: The endpoint response is defined manually. Specify the following:
        • HTTP Status Code: The HTTP Status code to handle. Select from the drop-down of status codes from the Open API specification. Alternatively, enter a custom code that conforms to the OpenAPI specification and is specific to your application.
        • Description (optional): Provide a useful description of the response, if desired.
        • Content Type:
          • application/json: The response body is in the form of JSON key-value pairs.
            • Response Message (optional): Specify the response message in the form of a valid JSON string.
          • String: The response body is in the form of a string.
            • Response Message (optional): Specify the response message as a string.
          • No Content: The response body contains no content.
  5. If you want this endpoint to have different caching requirements than those specified for the API by default, click the Caching tab under Create Endpoint, move the Inherit API Setting widget to the left, and specify the Cache Responses, as follows:
    • No: Endpoint responses are not cached.
    • Yes: Endpoint responses are cached according to the specified Time to Live (TTL) value in Seconds, Minutes, or Hours.
      • Note  Caches are accessible on the Resources page, under System > temp > cache.
  6. Click Create.

To Edit an Endpoint

On the Endpoints tab for your API, click the endpoint you want to edit. Note that changing the Response Type of an endpoint can also be handled on the Workspaces tab (described below).

If your saved edits affect the operation of the workspace assigned to the endpoint (for example, altering the Path), you must Re-download the workspace to FME Form. Alternatively, update the workspace in FME Form, then republish to FME Flow Data Virtualization, as follows:

  1. In FME Form, open the workspace.
  2. In the workspace Navigator, right-click on the Data Virtualization reader, and select Update Reader. This action retrieves the current endpoint settings from FME Flow Data Virtualization.
  3. Republish the workspace to FME Flow Data Virtualization.

To Purge the Cache of an Endpoint

You can purge the cache of a GET endpoint for which caching is enabled, either by default at the API level, or by itself.

On the Endpoints tab for your API, click the endpoint of the cache you want to purge. On the Caching tab, click Purge Cache.

Note  The cache is also purged when any of the following occurs:
  • FME Flow is restarted.
  • A workspace is republished to FME Flow Data Virtualization. Any cache associated with the previous version is cleared.
  • An update to the Time to Live setting that renders the cache older than the updated setting allows.

  • An update to the Path, Operation, or Inherit API setting.

To Remove an Endpoint

On the Endpoints tab for your API, click the ellipsis (...) beside the endpoint you want to remove, and select Delete.

To Create, Edit, or Remove Endpoint Tags

On the Endpoints tab for your API, click Manage Tags.

Generating and Managing Workspaces for Handling Endpoint Responses

In most cases, you will want your Data Virtualization API to reference one or more FME workspaces to handle endpoint responses. Each endpoint can reference a single workspace. As a first step for each workspace, you must generate a "starter" workspace. Once generated, you can download the workspace to FME Workbench, configure as desired, then republish the workspace to FME Flow Data Virtualization.

To Generate a Workspace

  1. On the Workspaces tab for your API, under Workspace Endpoints, select the endpoint you want to configure with the workspace, click Endpoint Actions, and select Generate Workspace.
  2. On the Generate Workspace dialog, enter a name for the workspace (if not specified explicitly, a .fmw extension is added automatically), and click Generate.

To Assign a Workspace to an Additional Endpoint

You can add endpoints to existing workspaces, as follows:

  1. On the Workspaces tab for your API, under Workspace Endpoints, select the endpoint for which you want a workspace assigned, click Endpoint Actions, and select Assign Workspace.
  2. On the Assign Workspace dialog, specify the workspace you want to assign to the endpoint, and click Assign.
  3. In FME Form, open the workspace. The workspace opens with the Data Virtualization reader auto-updated to include its newly-assigned endpoint.
  4. Republish the workspace to FME Flow Data Virtualization.

To Unassign a Workspace from an Endpoint

On the Workspaces tab for your API, under Workspace Endpoints, select the endpoint you want to unassign, click Endpoint Actions, and select Unassign Workspace.

To View, Download, or Remove Unassigned Workspaces

Click View Unused Workspaces. (This link is located directly following the description under the Workspace Endpoints heading.)

To Change the Response Type of an Endpoint

You can change how an endpoint response is handled: from a workspace to manually, or from manually to a workspace. This change can be handled on the Workspaces tab for your API.

To change an endpoint response from Workspace to Manual, select the endpoint under the Workspace Endpoints sub-tab, click Endpoint Actions, and select Edit Response Type.

To change an endpoint response from Manual to Workspace, select the endpoint under the Manual Endpoints sub-tab and click Edit Response Type.

To Know the Status of a Workspace Endpoint

The Workspace Endpoints tab provides information about the workspaces that are assigned to the endpoints of your API. Each endpoint lists metadata such as associated tags, date of last update, and assigned workspace. The Status column shows one of the following settings:

  • Unassigned: A workspace should be assigned or generated for this endpoint.
  • Needs Updating: An endpoint has been added, removed, or modified in FME Flow and the workspace needs updating. See Coordinating Updates Between FME Flow and FME Form (below).
  • Needs Authoring: The generated workspace needs to be configured in FME Form and published to FME Flow Data Virtualization.
  • Assigned: The workspace has been published from FME Form to FME Flow Data Virtualization.

Coordinating Updates Between FME Flow and FME Form

If you make changes to endpoints that affect the operation of the workspaces to which they are assigned, you must ensure those changes are reflected in those workspaces in FME Form, then republish to FME Flow Data Virtualization.

Updates to Endpoint Metadata

If you make edits to an endpoint that affect the operation of the workspace assigned to it (for example, altering the Path), you must Re-download the workspace to FME Form. Alternatively, update the workspace in FME Form, then republish to FME Flow Data Virtualization, as follows:

  1. In FME Form, open the workspace.
  2. In the workspace Navigator, right-click on the Data Virtualization reader, and select Update Reader. This action retrieves the current endpoint settings from FME Flow Data Virtualization.
  3. Republish the workspace to FME Flow Data Virtualization.

Assigning Workspaces to Additional Endpoints

You can add endpoints to existing workspaces, as follows:

  1. On the Workspaces tab for your API, under Workspace Endpoints, select the endpoint for which you want a workspace assigned, click Endpoint Actions, and select Assign Workspace.
  2. On the Assign Workspace dialog, specify the workspace you want to assign to the endpoint, and click Assign.
  3. In FME Form, open the workspace. The workspace opens with the Data Virtualization reader auto-updated to include its newly-assigned endpoint.
  4. Republish the workspace to FME Flow Data Virtualization.