Dynamic choices using Scripted Selection

Note

This functionality is new to FME 2024.

The Scripted Selection parameter type allows users to select from a dynamic list of options based on a Python callback. Applications of this parameter type include:

• Presenting choices that depend on the values of other parameters
• Presenting choices that originate from a remote source, such as a REST API
• Representing a remote filesystem

Tip

Use the Choice parameter type for cases where the list of choices is fixed and can be hardcoded into the parameter definition.

Overview

At a high level, a Scripted Selection parameter works by instantiating a Python class specified in its configuration, and then calling a method on that class to get the list of choices to present to the user. The Scripted Selection can be configured to include the values of other parameters in these calls, as well as constant key-values.

sequenceDiagram
    FME->>Python: Find ScriptedSelectionCallback implementation
    Python->>FME: Return instance
    FME->>Python: Call get_container_contents()
    Python->>FME: Return ContainerContentResponse
    loop when UI wants next page and pagination info exists
        FME->>Python: Call get_container_contents() with pagination info
        Python->>FME: Return ContainerContentResponse
    end

Scripted Selections have two types of choices:

• Containers are choices that may have children. Users may browse into containers to see its contents, but may not select them directly unless the Scripted Selection is configured to allow it.
• Items are choices that cannot be browsed into.

Both Containers and Items must specify a name and an ID.

Depending on the application, your implementation may:

• Return Items and Containers, representing a hierarchy such as files and folders.
• Only return Items, as there is no need for Containers.
• Have choices where the name and ID are always the same.
• Implement a search function.

Implement a Scripted Selection

In this tutorial, you will use Scripted Selection parameters to prompt users to select a continent and a city within that continent. Cities are grouped by country and state to demonstrate how to represent hierarchies in a Scripted Selection. To keep things simple, the dataset is hardcoded in Python instead of calling an external service.

This tutorial assumes that you are already familiar with the basics of developing FME Packages.

Getting started

Set up a development environment as follows:

1. Create a new FME Package project using the basic transformer template (fme-packager init transformer). When prompted, specify DemoSelector as the transformer name.
2. Vendorize fmetools >= 0.8.0. This is the version that introduces the Scripted Selection API.

Edit transformer definition and transformer.py

Make the following changes to the files in the new project:

1. Open the transformer definition FMXJ in FME Transformer Designer and delete the First Name sample parameter. It won't be used for this tutorial.

2. Open transformer.py and replace the lines in TransformerImpl.receive_feature():

   python/fme_demoselector/src/fme_demoselector/transformer.py

   ...
   class TransformerImpl(FMEEnhancedTransformer):
      ...
      def receive_feature(self, feature: FMEFeature):
         """
         Receive an input feature.
         """
         self.params.set_all(feature)
         continent_id, city_id = None, None
         try:
            continent_id = self.params.get("CONTINENT")
            city_id = self.params.get("CITY")
         except KeyError:
            pass  # Parameter not defined yet.
         feature.setAttribute(
            "message", 
            f"You picked City {city_id} in Continent {continent_id}"
         )
         self.pyoutput(feature)
   

   The CONTINENT and CITY parameters will be defined later.

Create select.py

Create select.py in the same folder as transformer.py. This will be the Python file that we will be working on for the rest of this tutorial. We will implement the Python callbacks for the Scripted Selection parameters in this file.

In select.py, define a variable called SAMPLE_DATA and assign it this JSON dataset. This dataset will be used to populate the Scripted Selections. Also define DATA_BY_ID to map all the IDs in the data to their corresponding objects. This simplifies the code later.

python/fme_demoselector/src/fme_demoselector/select.py

SAMPLE_DATA = [...]
# To simplify lookup, build a mapping of ID to country/state/city.
# Cast the IDs to strings because FME passes them back as strings.
DATA_BY_ID = {}
for continent in SAMPLE_DATA:
    DATA_BY_ID[str(continent["id"])] = continent
    for country in continent["countries"]:
        DATA_BY_ID[str(country["id"])] = country
        for state in country["states"]:
            DATA_BY_ID[str(state["id"])] = state
            for city in state["cities"]:
                DATA_BY_ID[str(city["id"])] = city

Continent parameter

First, we will create a Scripted Selection parameter for selecting the continent.

1. Open the transformer definition FMXJ in FME Transformer Designer.
2. Add a New Scripted Selection parameter from the dropdown menu. Give it the following configuration:

Setting	Value
Parameter Identifier	___XF_CONTINENT
Prompt	Continent
Python Entry Point	fme_demoselector.select.ContinentCallback
Allow Multiple Selection	No
Items Have Hierarchy	No
Allow Root Selection	No

Implement the Python callback

The scripted_selection module added in fmetools 0.8.0 provides the framework for writing the Python callback for a Scripted Selection parameter. It defines the interface that the callback must implement, and the schema of the data returned to FME.

Open select.py and create a class called ContinentCallback that extends fmetools.scripted_selection.ScriptedSelectionCallback. This class must implement the get_container_contents() method, which returns a ContainerContentResponse object. ContainerContentResponse contains a list of Item objects to be shown in the Scripted Selection dialog.

python/fme_demoselector/src/fme_demoselector/select.py

from typing import Optional
from ._vendor.fmetools.scripted_selection import (
   ScriptedSelectionCallback, ContainerContentResponse, Item
)
...
class ContinentCallback(ScriptedSelectionCallback):
    def get_container_contents(
        self, *, container_id: Optional[str] = None, limit: Optional[int] = None,
        query: Optional[str] = None, **kwargs
    ) -> ContainerContentResponse:
        items = []
        # Root level of SAMPLE_DATA is continents.
        for entry in SAMPLE_DATA:
            items.append(Item(
                id=str(entry["id"]), 
                name=entry["continent"],
                is_container=False
            ))
        return ContainerContentResponse(items)

Note that because this Scripted Selection is dedicated to selecting a continent, it has been configured to not allow browsing into Containers (Items Have Hierarchy = No). In addition, the Item objects returned by the Python callback have is_container set to False, despite each continent having child content.

Preview in FME Workbench

Save your changes, build, and install the package in FME Workbench. Then add the DemoSelector transformer to a workspace. Its parameters appear as follows:

Click "..." to open the Scripted Selection dialog, which presents the two continent options in the sample data:

After selecting one, run the workspace to see the message attribute on the output feature. If you picked "North America", the message should be You picked City None in Continent 1. Notice that the parameter value is the ID of the continent, not its name.

Enable debug logging

To help diagnose issues with the Python callback, enable debug logging in FME. When debug logging is enabled, exceptions raised from the Python callback will include the full error message and stack trace in the error dialog.

Debug logging is toggled through FME Options > Translation > Log Message Filter > Log Debug.

City parameter

Next, we will create a Scripted Selection parameter for selecting the city. Its implementation will be more complex than the continent parameter above, as its results depend on the value of the continent parameter as well as any country/state the user is browsing into.

Return to FME Transformer Designer and add a New Scripted Selection parameter. Give it the following configuration:

Setting	Value
Parameter Identifier	___XF_CITY
Prompt	City
Python Entry Point	fme_demoselector.select.CityCallback
Input Parameters	Enable ___XF_CONTINENT
Allow Multiple Selection	No
Items Have Hierarchy	Yes
Allow Root Selection	No

Implement the Python callback

Return to select.py and create the CityCallback class. Refer to the comments in the code below for the various cases to handle.

python/fme_demoselector/src/fme_demoselector/select.py

from fmeobjects import FMEException
...
class CityCallback(ScriptedSelectionCallback):
    def get_container_contents(
        self, *, container_id: Optional[str] = None, limit: Optional[int] = None,
        query: Optional[str] = None, **kwargs
    ) -> ContainerContentResponse:
        # Get Input Parameter value for the selected continent.
        continent_key = kwargs.get("___XF_CONTINENT")
        if not continent_key:
            raise FMEException("Pick a continent first")

        # Look for a matching continent.
        selected_continent = DATA_BY_ID.get(continent_key)
        if not selected_continent or "countries" not in selected_continent:
            raise FMEException(f"Continent {continent_key} not found")

        # If no container specified, then this is the root level.
        # Return the list of countries in the continent.
        if not container_id:
            items = []
            for country in selected_continent["countries"]:
                items.append(Item(
                    id=str(country["id"]),
                    name=country["country"],
                    is_container=True
                ))
            return ContainerContentResponse(items)

        # container_id specified.
        # Return values vary depending on whether it's a country or state.
        selected_element = DATA_BY_ID.get(container_id)
        if not selected_element:
            # Unrecognized ID.
            raise FMEException(f"Element {container_id} not found")
        if "states" in selected_element:
            # Selected a country. Return its states.
            items = []
            for state in selected_element["states"]:
                items.append(Item(
                    id=str(state["id"]),
                    name=state["state"],
                    is_container=True
                ))
            return ContainerContentResponse(items)
        if "cities" in selected_element:
            # Selected a state. Return its cities.
            # Cities are the final selection for this parameter.
            # The selected city's ID will be used as the parameter value.
            items = []
            for city in selected_element["cities"]:
                items.append(Item(
                    id=str(city["id"]),
                    name=city["city"],
                    is_container=False
                ))
            return ContainerContentResponse(items)
        if "city" in selected_element:
            # This shouldn't happen, as cities are returned with is_container=False.
            raise FMEException(f"Cannot browse into city '{selected_element['city']}'")

        # Not a country, state, or city.
        raise FMEException(f"Element {container_id} is invalid")

In summary, CityCallback does the following:

1. Get the value of the CONTINENT parameter. This value is provided to the callback because it was configured as an Input Parameter.
2. Resolve CONTINENT to a continent object in the sample data.
3. If no container ID is specified, return the list of countries in the selected continent.
4. If the container ID corresponds to a country, return its states.
5. If the container ID corresponds to a state, return its cities.
6. Handle error cases where the container ID is invalid, maps to an unrecognized type of object, or maps to an object that doesn't have any children (a city in this example).

Using the same Python callback for different Scripted Selections

Instead of using a different Python class for the Continent and City parameters, it's possible to use the same class for both. This can be useful when the Python implementations only differ slightly.

To do this, just set the same Python Entry Point for both parameters. The Python callback can then return different results depending on the input parameters. For instance, if a value for ___XF_CONTINENT is provided, the callback would return the countries/states/cities for that continent. Otherwise, it would return the continents.

Alternatively, the Scripted Selection could have constant parameters configured under Advanced > Input Constants. For instance, the Continent parameter specifies an Input Constants with a key value_type and value continents, while the City parameter specifies an Input Constants also with a key value_type but with a value of cities. The Python callback can then check the value of value_type to know what to return.

Preview in FME Workbench

Rebuild and reinstall the package. Add a new DemoSelector transformer to the workspace. The new City parameter is visible:

With "North America" selected as the continent, clicking "..." for the City parameter shows a browser where you can select a country, state, and then a city:

Notice that the display value for the selected city is a path-like representation of the hierarchy: "/Canada/Ontario/Ottawa":

However, in the Python code, the parameter value is the ID of the city ("14"). When you run the workspace, the message attribute should be You picked City 14 in Continent 1.

Pagination

The get_container_contents() method on the Python callback includes the limit keyword argument. FME may provide a value for this argument to suggest a limit for the number of items to return. Implementations may choose to respect this limit, but are not required to. This value can be useful in contexts where API calls also expose a limit parameter.

If there is a subsequent page of results, the callback's return value must include the arguments needed to fetch the next page. To provide pagination arguments:

1. Create a PaginationInfo object, with args set to a dict of pagination arguments. These arguments are included as keyword arguments when get_container_contents() is called to fetch the next page. Typical values are a page number, index offset, or next page token.
2. Set ContainerContentResponse.pagination to the PaginationInfo object before returning it.
3. If there are no more pages, set ContainerContentResponse.pagination to None.

When the Scripted Selection is ready to load the next page, it will call get_container_contents() with the additional pagination arguments.

The following is a sample implementation that paginates a list of numbers:

from typing import Optional

from ._vendor.fmetools.scripted_selection import (
    ScriptedSelectionCallback, ContainerContentResponse,
    Item, PaginationInfo
)

DATA = list(range(1000))  # Mock list of items


class Callback(ScriptedSelectionCallback):

    def get_container_contents(
        self, *, container_id: Optional[str] = None, limit: Optional[int] = None,
        query: Optional[str] = None, **kwargs
    ) -> ContainerContentResponse:
        # 'offset' is the pagination argument in this example.
        # If it's not present then assume 0, as in the first page.
        offset = kwargs.get("offset") or 0

        # Assume a reasonable page size when limit is not specified.
        limit = limit or 25

        # The subset of data to return for this request.
        page = DATA[offset:offset + limit]

        # If there's a next page, set the 'offset' pagination argument
        # and include it in the response.
        # Also propagate the limit arg because FME only provides it for the first call.
        pagination_info = None
        if offset + len(page) < len(DATA):
            pagination_info = PaginationInfo(dict(
                offset=offset + len(page),
                limit=limit,
            ))

        # Return list of mock items, and include the pagination info, if any.
        return ContainerContentResponse(
            [Item(str(i), f"Item {i}", is_container=False) for i in page],
            pagination_info
        )

Enable search

Scripted Selections may optionally support search. To enable search:

1. Enable Advanced > Supports Search in the Scripted Selection configuration. This adds the search field to the parameter's selection window.
2. In the Python callback, update get_container_contents() to use the query keyword argument. When the user enters a value in the search field, the value is passed to the callback via this argument. If search is not enabled or not in use, its value is None.

It's up to the implementation to decide how to handle the search string. FME does not expect any particular syntax.

Search results are returned the same way as regular results.

Allow direct value editing

In some cases, in addition to selecting from the list of choices, it may be desirable to allow users to directly enter a value for a Scripted Selection field.

To enable direct value editing:

• Enable Advanced > Name Matches Identifier in the Scripted Selection configuration. This allows the field to be edited directly.
• Disable Selection > Items Have Hierarchy in the Scripted Selection configuration. This makes the field's display value match the name of the selected Item.
• In the Python callback, only return Items with the same value for name and id.