FME Flow: 2025.0
Data Virtualization
With just basic knowledge of RESTful coding conventions, you can create data virtualization APIs that harness the power of FME workspaces to access and transform data from HTTP endpoints, while conforming to the OpenAPI standard.
The Data Virtualization page lists the APIs that are created. You can:
- Filter the list by title, namespace, and/or owner.
- Remove one or more APIs. Select its corresponding checkbox, click Actions, and select Remove.
- View an aggregated log file of data virtualization APIs. Select one or more APIs and click View Log File (requires Access and List permission to the Logs folder in Resources).
How It Works
The easy-to-use interface of FME Flow Data Virtualization gives you all the tools you need to define endpoints and configure them with workspace templates. Once your endpoints and templates are configured, download the templates to FME Workbench and configure your data access and transformation requirements. Your API is complete when you republish your finished workspaces to FME Flow Data Virtualization.
Getting Started
Follow these steps to create a data virtualization API:
- Create the API: Give the API a name, namespace, and repository for workspaces. Configure initial security, caching, and provide other metadata. See Creating and Managing Data Virtualization APIs, below.
- Configure API functionality: Define endpoints and configure with workspace templates. If desired, define reusable data property schemas for endpoints. Adjust security and metadata as desired. See Configuring API Functionality, below.
- Download workspace templates to FME Workbench, configure data access and transformation requirements, and republish to FME Flow Data Virtualization. These steps are handled in FME Workbench. For more information, see the following topics in the FME Workbench help:
Creating and Managing Data Virtualization APIs
To create an API directly in FME Flow, click Create. To import an API that you can tailor for FME Flow Data Virtualization, click Import. (If there are no previously-created APIs to display, you can also click Create API or Import OpenAPI under Get Started with APIs.)
On the Create API page or Import dialog, complete the following fields. When finished, click Create on the Create API page, or Upload on the Import dialog.
- API Title: (for APIs created directly in FME Flow only) Give the API a title.
- Namespace: Provide a namespace that is unique to the API. The namespace is used to route requests to this specific API.
- Choose a .json file: (for APIs imported to FME Flow only) Drop or select the .json file that comprises the API you want to import.
- Repository: Name of the repository to hold workspaces that are configured to process endpoint responses for this API. If desired, update the default name. After the API is created, this name cannot be changed.
For APIs created directly in FME Flow, the following additional fields may be completed. For all APIs created in FME Flow, these properties may be configured later. See "To Edit a Data Virtualization API", below.
- Version: This field autopopulates to track versions of the API.
- Summary: (Optional) Provide a summary description of the API.
- Description: (Optional) Provide a detailed description of the API.
- Job Queue: (Optional)
- Name: (Optional) Assign a contact person or organization for the API.
- URL: (Optional) A URL address that points to the contact Name.
- Email: (Optional) Email address for the contact Name.
- Access Level: (Optional)
- Authenticated: The endpoints of this API are accessible only to the Roles and Users specified under Allowed Roles and Allowed Users (below).
- Unauthenticated: The endpoints of this API do not require authentication to use.
Note You can update this setting later on the Security tab of the API after it is created. Additionally, when you create individual endpoints, you can configure security that overrides access at the API level. - Allowed Roles: (Optional) If Access Level is Authenticated, specify the roles to grant access to this API's endpoints.
- Allowed Users: (Optional) If Access Level is Authenticated, specify the users to grant access to this API's endpoints.
- Cache Responses: Specify default caching requirements for workspaces that respond to GET operations of the API's endpoints.
- No: Endpoint responses are not cached.
- Yes: Endpoint responses are cached according to the specified Time to Live (TTL) value in Seconds, Minutes, or Hours.
Click on an API to open it.
To edit the API Title, Namespace or Version, click the edit (pencil) icon beside the API Title.
To edit other properties, open the corresponding tab (Endpoints, Schemas, Security, API Details), and click the edit (pencil) icon corresponding to the section you want to edit. On the Security tab, you can also edit security for specific endpoints. Click the ellipsis (...) under ACTIONS corresponding to the endpoint you want to edit, and select Edit Security.
You may want to configure any URLs that will interact with your API for CORS. For more information, see Cross-Origin Resource Sharing.
You can download the OpenAPI .json specification file of the API. From the Data Virtualization home page, select its corresponding checkbox, click Actions, and select Export. You can also click Actions and select Export from the page of the API you opened.
When you create an API, documentation for its endpoints and schemas is also created. To test an endpoint, expand it and click Try it out.
To view API documentation, do any of the following:
- From the Data Virtualization home page, select the checkbox for the API, click Actions, and select View Documentation.
- From the page of the API you opened, click Actions and select View Documentation.
- On the API Details tab of the API, click View Documentation (under Description).
Data caches from API execution are available depending on the Cache Responses configuration of the API.
Caches are accessible on the Resources page, under System > temp > cache.
If caching is enabled for the API overall, or for any of its individual endpoints, you can purge all caches, as follows:
- Click on the API to open it.
- Open the API Details tab.
- Under Caching, click Purge Cache.
- On the Purge Cache dialog, click Purge.
Configuring API Functionality
Most of the functionality for your API is configured in the Endpoints and Workspaces tabs. The Schemas, Security, and API Details tabs provide additional ways to control and configure your API.
- Endpoints: Define the HTTP methods for receiving and processing requests for data.
- Workspaces: Generate and manage workspaces that handle endpoint responses.
- Schemas: Define reusable data property configurations for endpoints.
- Security: Edit who can access the API and its individual endpoints.
- API Details: Edit API metadata and caching.
Creating and Managing Schemas
Use the schemas tab to create schemas for defining data properties of HTTP requests in your endpoints, including the name of the data point, data type, and if the data is required. Schemas make it easy to reuse these data points across endpoints.
To Create a Schema
- On the Schemas tab for your API, click Create.
- On the Create Schema pane, provide a Name and Description (optional) for the schema.
- To add a data property to the schema, click Create beside Properties.
- Specify a Name, Type of data (optional), and if the data is Required. The specified Type (String, Number, Integer, or Boolean) functions according to its respective OpenAPI specification. If not specified, the schema accepts any type.
- Click Save.
To Edit a Schema
On the Schemas tab for your API, click the schema name you want to edit.
To remove a property, click the remove icon (trash can) beside the property.
To Remove a Schema
On the Schemas tab for your API, click the ellipsis (...) beside the schema you want to remove, and select Delete.
Creating and Managing Endpoints
HTTP endpoints specify how your API receives and processes requests for data.
To Create an Endpoint
- On the Endpoints tab for your API, click Create.
- On the Details tab under CREATE ENDPOINT, provide the following:
- Path: The URL for the endpoint.
- Operation: Specify the operation of the endpoint: GET, POST, PUT, or DELETE.
- Summary (optional): Provide a useful summary of the endpoint's operation, if desired.
- Description (optional): Provide a more detailed summary of the endpoint's operation, if desired.
- Tags (optional): Assign one or more tags to the endpoint, if desired. Tags are used to organize endpoints into meaningful groups on the Endpoints tab, based on your business rules. To create a tag before assigning it, click Create Tag.
- Endpoint Security: If you want access to the endpoint to be restricted under the same terms as those specified on the Security tab of the API, enable Inherit API Setting. To configure access separately for this endpoint, disable this setting, and specify:
- Access Level: (Optional)
- Allowed Roles: (Optional) If Access Level is Authenticated, specify the roles to grant access to this API's endpoints.
- Allowed Users: (Optional) If Access Level is Authenticated, specify the users to grant access to this API's endpoints.
Note On the Security tab of the API, you can always update security for specific endpoints after they are created.
- If Operation is POST or PUT, click the Request Body tab under CREATE ENDPOINT, and provide the following:
- Required: Specify if the request body is required to perform POST or PUT operation.
- Description (optional): Provide a useful description of the request body, if desired.
- Content Type:
- application/json: The request body is in the form of JSON key-value pairs.
- Input Type:
- Use Schema: The data properties of the request body are defined by the specified schema of the API.
- Create Properties: The data properties of the request body are defined manually. To create a property, click Create beside Properties, and specify a NAME, TYPE of data (optional), and if the data is REQUIRED. The specified TYPE (String, Number, Integer, or Boolean) functions according to its respective OpenAPI specification. If not specified, the property accepts any type.
- Click the Response tab under CREATE ENDPOINT, and specify the following:
- Response Type:
- Workspace: Use an FMEworkspace to handle the endpoint response. If a workspace template is not yet generated for this endpoint, click Save to save the current endpoint configuration and proceed to Generating and Managing Workspaces for Handling Endpoint Responses. When finished, return here and complete this step.
If the workspace is configured to handle specific status codes, click Add Status Code and specify:
- HTTP Status Code: The HTTP Status code to handle. Select from the drop-down of status codes from the Open API specification. Alternatively, enter a custom code that conforms to the OpenAPI specification and is specific to your application.
- Description (optional): Provide a useful description of the response, if desired.
- Content Type:
- application/json: The response body is in the form of JSON key-value pairs.
- Input Type:
- Use Schema: The data properties of the response body are defined by the specified schema of the API.
- Create Properties: The data properties of the response body are defined manually. To create a property, click Create beside Properties, and specify a NAME, TYPE of data (optional), and if the data is REQUIRED. The specified TYPE (String, Number, Integer, or Boolean) functions according to its respective OpenAPI specification. If not specified, the property accepts any type.
- Input Type:
- String: The response body is in the form of a string.
- application/json: The response body is in the form of JSON key-value pairs.
- Manual: The endpoint response is defined manually. Specify the following:
- HTTP Status Code: The HTTP Status code to handle. Select from the drop-down of status codes from the Open API specification. Alternatively, enter a custom code that conforms to the OpenAPI specification and is specific to your application.
- Description (optional): Provide a useful description of the response, if desired.
- Content Type:
- application/json: The response body is in the form of JSON key-value pairs.
- Response Message (optional): Specify the response message in the form of a valid JSON string.
- String: The response body is in the form of a string.
- Response Message (optional): Specify the response message as a string.
- No Content: The response body contains no content.
- application/json: The response body is in the form of JSON key-value pairs.
- Workspace: Use an FMEworkspace to handle the endpoint response. If a workspace template is not yet generated for this endpoint, click Save to save the current endpoint configuration and proceed to Generating and Managing Workspaces for Handling Endpoint Responses. When finished, return here and complete this step.
- Response Type:
- If you want this endpoint to have different caching requirements than those specified for the API by default, click the Caching tab under Create Endpoint, move the Inherit API Setting widget to the left, and specify the Cache Responses, as follows:
- No: Endpoint responses are not cached.
- Yes: Endpoint responses are cached according to the specified Time to Live (TTL) value in Seconds, Minutes, or Hours.
- Click Create.
To Edit an Endpoint
On the Endpoints tab for your API, click the endpoint you want to edit. Note that changing the Response Type of an endpoint can also be handled on the Workspaces tab (described below).
To Access Caches of an Endpoint
Data caches from endpoint execution are available depending on its Caching configuration.
Caches are accessible on the Resources page, under System > temp > cache.
To Purge the Cache of an Endpoint
You can purge the cache of a GET endpoint for which caching is enabled, either by default at the API level, or by itself.
On the Endpoints tab for your API, click the endpoint of the cache you want to purge. On the Caching tab, click Purge Cache.
To Remove an Endpoint
On the Endpoints tab for your API, click the ellipsis (...) beside the endpoint you want to remove, and select Delete.
To Create, Edit, or Remove Endpoint Tags
On the Endpoints tab for your API, click Manage Tags.
Generating and Managing Workspaces for Handling Endpoint Responses
In most cases, you will want your data virtualization API to reference one or more FME workspaces to handle endpoint responses. Each endpoint can reference a single workspace. As a first step for each workspace, you must generate a workspace template. Once generated, you can download the template to FME Workbench, configure as desired, then republish the workspace to FME Flow Data Virtualization.
To Generate a Workspace
- On the Workspaces tab for your API, under Workspace Endpoints, select the endpoint you want to configure with the workspace, and click Generate Workspace.
- On the Generate Workspace dialog, enter a name for the workspace (if not specified explicitly, a .fmw extension is added automatically), and click Generate.
To Unassign a Workspace from an Endpoint, or Reassign
On the Workspaces tab for your API, under Workspace Endpoints, select the endpoint you want to unassign, and click Unassign Workspace.
To reassign the workspace to the endpoint, click Reassign beside the endpoint.
To Change the Response Type of an Endpoint
You can change how an endpoint response is handled: from a workspace to manually, or from manually to a workspace. This change can be handled on the Workspaces tab for your API. To change an endpoint response from Workspace to Manual, select the endpoint under the Workspace Endpoints sub-tab and click Edit Response Type. To change an endpoint response from Manual to Workspace, select the endpoint under the Manual Endpoints sub-tab and click Edit Response Type.