Skip to content

Build a weather forecast package

In this guide, you will build on top of the basics from the Hello World guide and create an FME Package containing a transformer named WeatherFetcher, which gets weather data from a third-party REST API.

The WeatherFetcher transformer uses the OpenWeather API, which requires an API key. Sign up for an account at https://home.openweathermap.org/users/sign_up.

This guide will take you through multiple iterations of the WeatherFetcher transformer:

  1. An initial transformer with a few simple input parameters, and sets the current weather condition as attributes on the output feature.
  2. Adding an option to get the latitude and longitude from the input feature's geometry.
  3. Replacing the API Key parameter by using Web Services instead.

Each iteration increases the complexity of the transformer while introducing new concepts and features that will help you develop transformers of your own.

Getting started

Start by creating a new FME Package project using the fme-packager transformer template:

(venv)> fme-packager init transformer

When prompted, provide the following values:

  • package_uid: weather
  • transformer_name: WeatherFetcher

Accept the default value for all the other prompts.

This creates a folder named weather, which holds an FME Package project that contains a transformer named WeatherFetcher.

We will be making many edits to this FME Package, so get ready by setting up your development workflow using the steps described in the Hello World guide. To avoid conflicts, each FME Package project should have its own Python virtualenv.

FME 2023 compatibility

This tutorial is designed for transformers which do not need to support FME 2023.

For transformers which support FME 2023,

  1. Ensure that minimum_supported_fme was set to 2023 when creating the FME Package project.
  2. In the Python implementation, calls to pyoutput() should omit the output_tag keyword argument.

Initial implementation

Now we will create an initial implementation of the WeatherFetcher transformer. This transformer will have a few simple input parameters, and will set the current weather condition as attributes on the output feature. The weather data will be obtained from the current weather data endpoint of the OpenWeather API.

In this section you will learn how to:

  • Define parameters on a transformer
  • Read parameter values
  • Make HTTP requests
  • Set output attributes and expose them in FME Workbench
  • Reject features when there is an error

Define transformer parameters

We need to update the WeatherFetcher transformer definition with parameters that correspond to arguments in the API request: API Key, Latitude, Longitude, Units, and Language. Open WeatherFetcher.fmxj in the FME Transformer Designer.

Initial state of WeatherFetcher

First, delete the initial parameters defined in the template. Select the Parameters group and click the Delete button in the toolbar, or press Del. Repeat this with the First Name parameter.

Delete initial parameters

Next, click the Parameter icon in the toolbar. This opens a menu of supported parameter types. We will start by adding an Authorization parameter group, and an API Key parameter inside it:

  1. Add a Group Box.
  2. With the new Group Box selected, switch to the Configuration pane and change the Prompt from "Parameters" to "Authorization".
  3. With the Group Box still selected, click the Parameter icon in the toolbar again, and add a Password parameter. This puts the new parameter inside the Group Box.
  4. Make the following changes to the Password parameter:
    • Prompt: API Key
    • Parameter Identifier: ___XF_API_KEY

WeatherFetcher with authorization group added

Repeat this process to add the remaining parameters:

Group Parameter Prompt Type Parameter Identifier Configuration
Coordinates Latitude Number ___XF_LATITUDE Numeric Precision: Float
Coordinates Longitude Number ___XF_LONGITUDE Numeric Precision: Float
Options Units Choice ___XF_UNITS Choice Values standard, metric, and imperial, with Display Values "Kelvin", "C", and "F" respectively.
Default Value: Kelvin
Options Language Text ___XF_LANGUAGE Required: No

Save the changes to the transformer definition file. The parameters section should now look like this:

New WeatherFetcher parameters in FME Transformer Designer

In FME Workbench, the WeatherFetcher parameters dialog now looks like this:

Default state

Fields populated with valid values

For more information about defining parameters and parameter configuration, see FME Transformer Designer > Configuring a Transformer.

Get parameter values from input feature

We are now ready to get the input parameters in the Python implementation of the transformer. Open transformer.py in the package's Python module and update the receive_feature() method:

python\fme_weatherfetcher\src\fme_weatherfetcher\transformer.py
"""
example.weather.WeatherFetcher implementation.
"""
from fmeobjects import FMEFeature
from ._vendor.fmetools.plugins import FMEEnhancedTransformer
from ._vendor.fmetools.paramparsing import TransformerParameterParser


class TransformerImpl(FMEEnhancedTransformer):
    """
    This is the Python implementation of the transformer.
    Each instance of the transformer in the workspace has an instance of this class.
    """

    params: TransformerParameterParser
    version: int

    def setup(self, first_feature: FMEFeature):
        """
        Initialization steps based the first feature received by the transformer.
        """
        super().setup(first_feature)
        # Get transformer version from internal attribute on first feature,
        # and load its parameter definitions.
        # Note: TransformerParameterParser requires >=b24145 when running on FME Flow.
        self.version = int(first_feature.getAttribute("___XF_VERSION"))
        self.params = TransformerParameterParser(
            "example.weather.WeatherFetcher",
            version=self.version,
        )

    def receive_feature(self, feature: FMEFeature):
        """
        Receive an input feature.
        """
        # Collect all input parameters
        self.params.set_all(feature)
        api_key = self.params.get("API_KEY")
        latitude = self.params.get("LATITUDE")
        longitude = self.params.get("LONGITUDE")
        units = self.params.get("UNITS")
        language = self.params.get("LANGUAGE")

        self.pyoutput(feature, output_tag="Output")

TransformerParameterParser on FME Flow

Instantiating TransformerParameterParser on FME Flow requires FME Flow b24145 or newer. Instantiating the class on older versions of FME Flow will raise an exception.

Make an HTTP request and set output attributes

To make HTTP requests, use fmetools.http.FMERequestsSession. It is built on top of the Requests library and provides integrations with FME, such as Custom Proxy Map support.

Now we will use FMERequestsSession to call the OpenWeather API to get the current weather conditions. Then, we will take a few values from the response and set them as attributes on the output feature.

Update transformer.py as follows:

python\fme_weatherfetcher\src\fme_weatherfetcher\transformer.py
from ._vendor.fmetools.http import FMERequestsSession
from ._vendor.fmetools.features import set_attributes, set_attribute
...

class TransformerImpl(FMEEnhancedTransformer):
    ...
    def __init__(self):
        super(TransformerImpl, self).__init__()
        self.session = FMERequestsSession()

    def receive_feature(self, feature: FMEFeature):
        ...
        # API doc: https://openweathermap.org/current#one
        resp = self.session.get(
            "https://api.openweathermap.org/data/2.5/weather",
            params={
                "lat": latitude,
                "lon": longitude,
                "appid": api_key,
                "units": units,
                "lang": language,
            }
        )
        result = resp.json()

        for key, value in result["main"].items():
            set_attribute(feature, key, value)

        set_attributes(feature, {
            "description": result["weather"][0]["description"],
            "location_name": result["name"],
            "rain_1h_mm": result.get("rain", {}).get("1h"),
        })

        self.pyoutput(feature, output_tag="Output")

In this snippet of code, we first make a request to the OpenWeather API's current conditions endpoint. As part of the request, we pass in all the parameter values that were fetched in the previous section. The response is then parsed as JSON.

Next, we set some output attributes using values from the response:

  • The main key in the response contains information such as the current temperature, daily high/low, and humidity. These attributes are set on features one at a time using the fmetools.features.set_attribute function.
  • The weather description, location name, and recent rainfall are set using the fmetools.features.set_attributes function. This function provides a convenient way to set multiple attributes at once by passing in a dictionary.
  • If there is no rainfall data in the response, the corresponding attribute is set to null.

Warning

While FMEFeature has methods for working with attributes, these methods need extra care around null values. The attribute getters and setters from fmetools should be used whenever possible, as they handle these details for you. Otherwise, refer to the fmeobjects documentation for FMEFeature.getAttribute() and FMEFeature.setAttribute() for details on working with null values.

The next step requires a valid key for the OpenWeather API. Sign up for an account at https://home.openweathermap.org/users/sign_up.

To see these changes in action, open FME Workbench, add a WeatherFetcher transformer, fill in its parameters with a valid OpenWeather API key and some coordinates, connect the output port to a Logger, and run the workspace.

Below is an example of what the logged output feature may look like for the weather conditions at the coordinates 49, -123.08:

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `WeatherFetcher_Output_LOGGED'
Attribute(64 bit unsigned integer): `_creation_instance' has value `0'
Attribute(string: UTF-8)          : `description' has value `scattered clouds'
Attribute(64 bit real)            : `feels_like' has value `9.76'
Attribute(string: UTF-8)          : `fme_feature_type' has value `Creator'
Attribute(string: UTF-8)          : `fme_geometry' has value `fme_undefined'
Attribute(string: UTF-8)          : `fme_type' has value `fme_no_geom'
Attribute(32 bit integer)         : `grnd_level' has value `1012'
Attribute(32 bit integer)         : `humidity' has value `62'
Attribute(string: UTF-8)          : `location_name' has value `Point Roberts'
Attribute(32 bit integer)         : `pressure' has value `1018'
Attribute(string: UTF-8)          : `rain_1h_mm' is <null>
Attribute(32 bit integer)         : `sea_level' has value `1018'
Attribute(64 bit real)            : `temp' has value `10.99'
Attribute(64 bit real)            : `temp_max' has value `13.73'
Attribute(64 bit real)            : `temp_min' has value `8.23'
Coordinate System: `'
Geometry Type: IFMENull
Coordinate Dimension: 2
===========================================================================

The output feature has all the attributes that were set, containing values from the OpenWeather API response.

Error handling and using the rejected port

When an error occurs in a transformer, it should output a rejected feature instead of a typical output feature. Rejected features are emitted from the <Rejected> port on the transformer instead of the Output port. This mechanism allows workspace authors to include error handling so that errors do not stop the entire translation.

To output a rejected feature, use the reject_feature() method instead of pyoutput() in your transformer implementation. Both of these methods are defined on the fmetools.plugins.FMEEnhancedTransformer base class.

One situation where the WeatherFetcher transformer should output a rejected feature is when the OpenWeather API returns an error response. One reason for the API to return an error is when the API key is invalid. The error response JSON looks like this:

Sample error response from the OpenWeather API
{
    "cod": 401,
    "message": "Invalid API key. Please see http://openweathermap.org/faq#error401 for more info."
}

Now we will update the code to handle error responses from the OpenWeather API, and convert them to a rejected feature:

python\fme_weatherfetcher\src\fme_weatherfetcher\transformer.py
...
class TransformerImpl(FMEEnhancedTransformer):
    ...
    def receive_feature(self, feature: FMEFeature):
        ...
        result = resp.json()

        if result["cod"] != 200:
            self.reject_feature(
                feature,
                code=f"CODE_{result['cod']}",
                message=result.get("message")
            )
            return
        ...

If the cod key in the response is not 200, then the response is an error. The feature is rejected, and given a code and message as the reason, which are set as attributes on the rejected feature. It is up to you to decide what the code and message should contain. Users of the transformer may rely on them for error handling.

Return to FME Workbench and edit the WeatherFetcher's parameters to have an invalid API key. When the workspace is run, the translation should fail due to the unhandled rejected feature. The log will also include the rejected feature:

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(64 bit unsigned integer): `_creation_instance' has value `0'
Attribute(string: UTF-8)          : `fme_feature_type' has value `Creator'
Attribute(string: UTF-8)          : `fme_geometry' has value `fme_undefined'
Attribute(string: UTF-8)          : `fme_rejection_code' has value `CODE_401'
Attribute(string: UTF-8)          : `fme_rejection_message' has value `Invalid API key. Please see http://openweathermap.org/faq#error401 for more info.'
Attribute(string: UTF-8)          : `fme_type' has value `fme_no_geom'
Coordinate System: `'
Geometry Type: IFMENull
Coordinate Dimension: 2
===========================================================================

Notice the fme_rejection_code and fme_rejection_message attributes, which correspond to the code and message arguments of reject_feature().

About rejected feature handling

By default, unhandled rejected features will fail the translation. However, this behavior is configurable in each workspace. For more information, refer to the documentation on Rejected Feature Handling.

Expose output attributes

On the FME Workbench canvas, notice that the transformer's output ports can be expanded to reveal a list of attributes. These are called exposed attributes, which means they appear in the Attribute Value picker of downstream transformers. Setting an attribute on output features in Python does not automatically expose that attribute. Instead, exposed attributes are set in the transformer definition file using the Attributes To Add configuration of each output port.

Return to the WeatherFetcher definition in FME Transformer Designer, and select the Output port. Notice the Attributes To Add configuration defines an output attribute named _greeting of type buffer (string):

WeatherFetcher in Transformer Designer showing output attribute

Now we will revise the transformer definition and expose some attributes we had set in the code. Remove _greeting and add description, location_name, and temp. As the first two attributes have string values, assign them the buffer type. As temp is a decimal value, assign it the real64 type.

New output attributes on WeatherFetcher

For brevity, the example above exposes only 3 attributes. In practice, every output attribute set in the Python code should also be declared as an exposed attribute in the transformer definition file.

After saving the updated transformer definition, restart FME Workbench and add the WeatherFetcher onto the canvas again. With the output attributes expanded, it should look something like this:

WeatherFetcher on canvas with output attributes shown

It is also possible for users to manually define exposed attributes using the AttributeExposer transformer. But this is not ideal and should not be necessary if transformers accurately define every output attribute as an exposed attribute.

Add option to get coordinates from input feature

Now we will extend WeatherFetcher by adding an option to specify the coordinates through the geometry of the input feature, instead of using the Longitude and Latitude parameters.

In this section you will learn how to:

  • Define a new version of a transformer
  • Enable/disable parameters depending on the value of another parameter
  • Work with geometry on input features
  • Read coordinates from FMEPoint geometry
  • Check for coordinate systems and reproject a feature to another coordinate system
  • Log messages to the FME log file

Define a new transformer version

Each transformer definition file defines all possible versions of its transformer. A new version of a transformer should be created whenever there are significant changes to its interface, such as the addition of a new required parameter. When editing a workspace containing an older version of a transformer or package, FME Workbench prompts users with the option to upgrade to the latest version.

Transformer versions are independent of package versions. While package updates must increase the package version defined in package.yml, they do not require defining a new transformer version.

As we are adding a new parameter that determines where the transformer will get its coordinates, and this parameter is required, it makes sense to define a new transformer version for this change.

To define a new version of the WeatherFetcher transformer, open it in FME Transformer Designer and select Edit > New Version. A dialog appears, noting that version 2 of the transformer will be created, and prompting for an optional change log entry:

New Version dialog

Click OK to create the new version. Notice that Version 2 is now selected in the toolbar. The version number also needs to be updated in the Execution Instruction Template and in the package metadata.

Open the Execution Instructions pane from the View menu. Then, with Version 2 of the transformer selected, edit the Execution Instruction Template. Change the line that sets the ___XF_VERSION attribute so it has a value of 2. This is the value passed to the Python code, which uses it to create a TransformerParameterParser for that version of the transformer.

Execution Instruction Template
# Put config values onto input feature as attributes.
FACTORY_DEF {*} TeeFactory
   FACTORY_NAME { $(XFORMER_NAME)_CATCHER }
   $(INPUT_LINES)
   OUTPUT { FEATURE_TYPE $(XFORMER_NAME)_READY
      @SupplyAttributes(___XF_VERSION, 2)
      $(FME_PARM_VAL_LIST)
   }
...

Next, update the package metadata to specify that Version 2 of WeatherFetcher is available:

package.yml
fpkg_version: 1
uid: weather
publisher_uid: example
name: My FME Package
description: A short description of my FME Package.
version: 0.1.0
minimum_fme_build: 24158
author:
  name: FME Lizard
package_content:
  transformers:
    - name: WeatherFetcher
      version: 2
  python_packages:
    - name: fme_weatherfetcher

When to define a new version of a transformer

Changes to a transformer definition do not always require defining a new transformer version. Generally, a new version should be defined when:

  • Adding or removing input/output ports
  • Changing the default value of a parameter
  • Adding a new required parameter that does not have a reasonable default value
  • Behavior has significantly changed under the same parameter values

Care should be taken to ensure backwards compatibility. Older versions of the transformer should continue to function as before.

Set up conditional transformer parameters

In this section, we will add a parameter that toggles whether to get coordinates from the Latitude and Longitude parameters, or a new ability to get coordinates from the input feature's geometry.

Select the Coordinates group in FME Transformer Designer, and add a new Choice parameter with the following configuration:

Setting Value
Parameter Identifier ___XF_COORD_SRC
Prompt Source
Choice Configuration Values params and geom, with Display values Parameters and Geometry respectively
Default Value Parameters

Next, we want the Latitude and Longitude parameters to be disabled when Source is set to Geometry. This is done through the Conditional Visibility configuration for each parameter. For both Latitude and Longitude, enable Conditional Visibility, and give it the following settings:

Setting Value
If ___XF_COORD_SRC Is Parameters
Then Show As Enabled
Else Show As Disabled

Save the new transformer definition. Then, in FME Workbench, observe how the coordinate parameters are enabled and disabled based on the value of Source:

Source is 'Parameters'

Source is 'Geometry'

Note

Changes to transformer files may not be reflected until FME Workbench is restarted or the Transformer Gallery is reloaded. To reload the Transformer Gallery, click the button on the bottom right corner of its pane. The Transformer Gallery pane can be enabled under View > Windows > Transformer Gallery.

Update code for conditional parameters

Next, the Python code needs to be updated to get the value of ___XF_COORD_SRC and change its logic accordingly. Open transformer.py and apply the following changes:

python\fme_weatherfetcher\src\fme_weatherfetcher\transformer.py
...
class TransformerImpl(FMEEnhancedTransformer):
    ...
    def receive_feature(self, feature: FMEFeature):
        """
        Receive an input feature.
        """
        # Collect all input parameters
        self.params.set_all(feature)
        api_key = self.params.get("API_KEY")
        units = self.params.get("UNITS")
        language = self.params.get("LANGUAGE")
        coord_src = self.params.get("COORD_SRC") or "params"

        if coord_src == "geom":
            geom = feature.getGeometry()
            # TODO: validation and error handling
            longitude, latitude, z = geom.getXYZ()
        else:
            latitude = self.params.get("LATITUDE")
            longitude = self.params.get("LONGITUDE")
        ...

These changes set up the transformer code to get the value of the new Source parameter. Then, if it is set to geom, it gets the latitude and longitude values from the input geometry instead of transformer parameters. We will expand on that logic in the next section. For now, take note of the following:

  • If the ___XF_COORD_SRC internal attribute is missing, fall back to a value of params, which means the transformer continues to get coordinates from transformer parameters, as before.
  • The values of ___XF_COORD_SRC are not the display values.
  • The internal attributes for latitude and longitude are only fetched when Source is not Geometry, or in other words, when these parameters are expected to contain a valid value. This avoids getting unnecessary parameters, as well as errors that may arise from getting the value of a disabled parameter.

Tip

Avoid fetching values for disabled transformer parameters. Doing so may raise an error or return an unusable result. The fetching of conditional parameter values should also be conditional on the parameter that enables it.

Transformer version compatibility

The Python transformer implementation must be backward compatible with older transformer versions. Here, that is accomplished by handling the possibility that the ___XF_COORD_SRC internal attribute may be missing, by falling back to the default value of params. This is because workspaces with version 1 of WeatherFetcher transformer will not have the new internal attribute added in version 2.

It is important to maintain backward compatibility, as users may upgrade to the latest version of a package, but not upgrade their workspaces to use that package's latest transformer version.

Working with input geometry

Now we will focus on the code for the case where the Source parameter is set to Geometry.

python\fme_weatherfetcher\src\fme_weatherfetcher\transformer.py
...
class TransformerImpl(FMEEnhancedTransformer):
    ...
    def receive_feature(self, feature: FMEFeature):
        ...
        if coord_src == "geom":
            geom = feature.getGeometry()
            # TODO: validation and error handling
            longitude, latitude, z = geom.getXYZ()
        ...

This is a simple implementation that gets the geometry of the input feature and assumes that it has a getXYZ() method, and that the X and Y coordinates it returns correspond to longitude and latitude, respectively. In practice getGeometry() can return any FMEGeometry object, and only FMEPoint has a getXYZ() method. If this code encounters anything that is not an FMEPoint, it will crash. In addition, the geometry may be in a different coordinate system, or no coordinate system at all.

Now we will handle these cases by updating the code to fill in the TODO:

python\fme_weatherfetcher\src\fme_weatherfetcher\transformer.py
from fmeobjects import FMEPoint
...
class TransformerImpl(FMEEnhancedTransformer):
    ...
    def receive_feature(self, feature: FMEFeature):
        ...
        if coord_src == "geom":
            geom = feature.getGeometry()
            if not isinstance(geom, FMEPoint):
                self.reject_feature(feature, code="BAD_GEOM", message="not a point")
                return
            if feature.getCoordSys() != "LL84":
                self.reject_feature(feature, code="NEED_LATLNG", message="not in LL84")
                return
            longitude, latitude, z = geom.getXYZ()
        ...

With these changes, the feature is rejected if the input geometry is not an FMEPoint, or if it is not in the LL84 coordinate system. Try this out by using a Creator to build invalid input geometry, and then running the translation.

Coordinate system reprojection and replacement

To reproject a feature to a different coordinate system, use FMEFeature.reproject(). Beware that reprojection can fail, such as when the source coordinate system cannot be reprojected to the target coordinate system, or if the target coordinate system is unrecognized. To replace the declared coordinate system without reprojection, use FMEFeature.setCoordSys().

As an exercise, try revising the transformer implementation so that it attempts to reproject features that are not in LL84, while handling the failure cases appropriately.

There are many geometry classes in FME, and some of them are fairly complex. For details, refer to the Python fmeobjects Geometries API reference. As an exercise, try extending WeatherFetcher to accept all FMEArea types (polygons, ellipses, donuts) by using the centroid of the geometry's bounding box.

Output log messages

Sometimes, it may be useful for your code to output messages in order to report progress, log debug information, or warn the user about something that is not severe enough to warrant failure or feature rejection. The FMEEnhancedTransformer class includes a log member for this purpose. It integrates with the FME log file and has the same interface as the Python standard library's logging.Logger.

Now we will show this in action by making the WeatherFetcher more lenient about the input feature's coordinate system. Specifically, we will change the transformer to assume that features without a coordinate system are already in LL84. Update the code as follows:

python\fme_weatherfetcher\src\fme_weatherfetcher\transformer.py
...
class TransformerImpl(FMEEnhancedTransformer):
    ...
    def receive_feature(self, feature: FMEFeature):
        ...
        if coord_src == "geom":
            ...
            if feature.getCoordSys() != "LL84":
                self.reject_feature(feature, code="NEED_LATLNG", message="not in LL84")
                return
            ...
python\fme_weatherfetcher\src\fme_weatherfetcher\transformer.py
...
class TransformerImpl(FMEEnhancedTransformer):
    ...
    def receive_feature(self, feature: FMEFeature):
        ...
        if coord_src == "geom":
            ...
            coordsys = feature.getCoordSys()
            if not coordsys:
                self.log.warning("Assuming LL84 for feature without coordsys")
            elif coordsys != "LL84":
                self.reject_feature(feature, code="NEED_LATLNG", message="not in LL84")
                return
            self.log.info("Getting coordinates from geometry...")
            ...

Now the transformer logs a message whenever it gets coordinates from the input geometry. When it encounters geometry without a coordinate system, it now logs a warning instead of rejecting the feature.

Run the translation again, but this time configure the input geometry to have no coordinate system. Both of the new log messages should appear in the FME Workbench log window:

Log window warning example

Tip

Debug-level messages will appear in the FME log if FME Workbench or the workspace has debug logging enabled. To enable debug logging in FME Workbench, go to Tools > FME Options > Translation > Log Message Filter and enable “Log Debug”.

FMERequestsSession automatically logs HTTP traffic when debug logging is enabled.

Use a Web Service to hold credentials

So far, the WeatherFetcher transformer has been set up such that users provide their OpenWeather API key using a password field parameter. This is a simple and straightforward approach, but has the downside of being saved as part of the workspace. As a result, when sharing workspaces with the WeatherFetcher transformer, users may also unintentionally share their API key. It also means that if there are many workspaces using the same API key, and that API key needs to be updated, then all those workspaces will need to be manually edited.

FME Web Services provides a centralized credential store that helps solve these problems. By adopting Web Services, parameter(s) that contain credentials are replaced with a reference to an entry in the FME Web Services credential store. These entries are called Named Connections.

In this section you will learn how to:

  • Define a Web Service for storing credentials
  • Include the Web Service in an FME Package
  • Add a Named Connection transformer parameter that uses a Web Service
  • Get a Named Connection in Python and retrieve credentials from it

Define a Web Service

A Web Service is an XML file that defines an authentication method, or a set of parameters needed to authenticate with a service. It could be as simple as defining a username and password field to use for HTTP Basic authentication, or as complex as an OAuth 2.0 desktop application authorization workflow.

For WeatherFetcher, all that is needed is a Token Web Service that defines a field to hold the OpenWeather API key. To create it, open FME Workbench and navigate to Tools > FME Options > Web Connections > Manage Services. Then open the menu on the bottom left and select Token Service.

Manage Web Services dialog

After selecting Token Service, the dialog changes to editing a New Web Service. Make the following changes to define a Web Service that we will eventually use in the WeatherFetcher:

  1. Web Service Name: Enter example.weather.OpenWeather. The name should reflect the API or service being connected to, and not the transformers that use it. The name must be fully qualified with the package Publisher UID and Package UID.
  2. Web Service Setup Instructions: The content of this field is shown to users when defining a Named Connection based on this Web Service. Provide a brief description of OpenWeather and where to sign up for an API key.
  3. Parameters: Define a new Password parameter. For Prompt, use “API Key”. Check the Required box. The Parameter Identifier is the mapping key used to retrieve the value in Python. Set it to API_KEY.
  4. Token Generation REST API: Uncheck this group. OpenWeather does not need it.
  5. API Call Parameters:
    • Placement: Query String
    • Token Item Key: appid

Once the Web Service is fully defined, it needs to be exported as a file.

  1. In the package project root, create a folder named web_services. The exported Web Service definition will be saved here.
  2. In the Manage Web Services dialog above, click the Export button at the bottom of the new Web Service definition.
  3. In the Export Web Service dialog, uncheck Include Credentials. In Export To Folder, select the web_services folder created in step 1.
  4. Click OK to export the Web Service.

This creates a file named example.weather.OpenWeather.xml that contains the Web Service definition.

Put the Web Service definition in the FME Package

The exported Web Service definition needs to be added to the FME Package manifest, so that it is included in the FPKG files created by fme-packager, and installed by FME Workbench.

  1. Rename example.weather.OpenWeather.xml to OpenWeather.xml. It is not necessary for the filename to be fully qualified.
  2. In package.yml, add a web_services key under package_content, as a list with a single element of OpenWeather.xml.
    package.yml
    fpkg_version: 1
uid: weather
publisher_uid: example
name: My FME Package
description: A short description of my FME Package.
version: 0.1.0
minimum_fme_build: 24158
author:
  name: FME Lizard
package_content:
  transformers:
    - name: WeatherFetcher
      version: 2
  python_packages:
    - name: fme_weatherfetcher
  web_services:
    - name: OpenWeather.xml

Now, verify that the Web Service is properly included in the FME Package. In the Manage Web Services dialog, remove any existing OpenWeather definition. After reinstalling the package, the OpenWeather definition reappears in the Manage Web Services dialog.

Updating or reinstalling a package will overwrite the previously installed Web Service definition. Web Service definitions are kept in the FME Connection Store database. This means that unlike other components of FME Packages, Web Service definitions are not installed into FME Workbench by a file copy.

Remember that if you are using symlinks in your package installation for development purposes, reinstalling the package will remove those symlinks and replace them with the package's files.

Note

Uninstalling an FME Package does not remove its included Web Service definition.

Replace the API Key parameter

Now we will update the transformer parameters and replace the API Key password field with a Named Connection field that references the new OpenWeather Web Service that we have defined.

Make the following changes in FME Transformer Designer:

  1. Define Version 3 of the transformer. As discussed in the earlier section about defining a new transformer version, it is best practice to define a new transformer version whenever there are significant changes to its parameters. Remember to also increment the version in package.yml and ___XF_VERSION in the Execution Instruction Template.
  2. Select the existing API Key parameter, and delete it.
  3. In the now-empty Authorization group, add a Web Connection parameter with the following configuration:
    • Parameter Identifier: ___XF_CONNECTION
    • Prompt: OpenWeather Connection
    • Supported Services: OpenWeather (example.weather). Note that only installed Web Services are shown, so make sure to first install the Web Service that was created in the previous section.

The transformer parameters now look like this:

WeatherFetcher parameters with connection parameter

Go ahead and create a new Named Connection for OpenWeather, and enter your API Key. Later, we will use this connection to make sure that everything still works.

OpenWeather connection definition dialog

Update code to get the API key from a Named Connection

Now that the package includes a Web Service definition for OpenWeather, and the API Key password field in the WeatherFetcher parameters has been replaced with a Named Connection picker, the Python code needs to be updated to use the Named Connection.

The value of the Named Connection parameter is the name of the selected Named Connection. The Python code needs to use this name to get the actual Named Connection object, which contains credentials. In the case of the OpenWeather Web Service, the credential is the API Key.

Update the transformer's Python code with the following changes. Take note of the typical considerations needed when implementing support for Named Connections.

python\fme_weatherfetcher\src\fme_weatherfetcher\transformer.py
...
from fmewebservices import FMENamedConnectionManager, FMETokenConnection


class TransformerImpl(FMEEnhancedTransformer):
    ...
    def receive_feature(self, feature: FMEFeature):
        self.params.set_all(feature)
        if self.version < 3:
            # Older versions have an API Key parameter only.
            api_key = self.params.get("API_KEY") # (1)
        else:
            conn_name = self.params.get("CONNECTION")
            mgr = FMENamedConnectionManager()
            conn = mgr.getNamedConnection(conn_name)  # (2) type: FMETokenConnection
            if not conn: # (3)
                self.reject_feature(
                    feature,
                    code="CONNECTION_NOT_FOUND",
                    message="Named connection not found"
                )
                return
            self.log.info("Using connection '%s'", conn_name) # (4)
            api_key = conn.getKeyValues()["API_KEY"] # (5)
            self.session.verify = conn.getVerifySslCertificate() # (6)

        # Collect remaining input parameters
        units = self.params.get("UNITS")
        ...
  1. To maintain compatibility with old workspaces, we keep recognizing the API_KEY parameter on previous transformer versions. It is important to only attempt to get the API_KEY parameter when on an old version. On Version 3, where the parameter has been removed, attempting to get its value will raise KeyError.
  2. The Named Connection object is retrieved by name, using fmewebservices.FMENamedConnectionManager.
  3. If there is no such connection for the given name, getNamedConnection() returns None. It is important to handle this case, as the parameter may reference a connection that was moved or deleted.
  4. It is helpful to log the Named Connection that is being used.
  5. In the case of the Token Service that was defined for OpenWeather, all the Named Connections derived from it are returned as instances of fmewebservices.FMETokenConnection. FMETokenConnection.getKeyValues() returns a mapping of parameters to values. As the OpenWeather Web Service only defines a single parameter with a Key Name of API_KEY, that is all it returns here.
  6. Each Token Connection includes a toggle for whether to verify HTTPS certificates. This setting is available through the getVerifySslCertificate() method. To obey this setting, its value is directly set to the verify property of the Requests Session.

Try out the new code by creating a workspace with the updated WeatherFetcher and selecting the Named Connection you had created in the previous section. Run the workspace, and you should see that the API key is retrieved from the Named Connection, and the request is successfully authenticated, just like with the API_KEY parameter before. Old workspaces should still continue to work because the code handles that scenario.

Prepare for release

The WeatherFetcher transformer is now fully implemented. Just a few more steps are needed prior to distribution.

Write user documentation

Now we will update the documentation, which is written in the Markdown format. This involves three files that are used in different contexts:

  • README.md: The package description shown on FME Hub.
  • transformers\WeatherFetcher.md: The description of the transformer shown in the Quick Add pane of FME Workbench.
  • help\WeatherFetcher.md: The main documentation for the transformer.

First, update the package description in README.md. The contents of this file are shown on FME Hub, where prospective users may decide whether the package meets their needs before downloading it. For instance, it could look like this:

README.md
# My FME Package (example.weather)

This package was made as part of a tutorial.
It provides the WeatherFetcher transformer,
which gets the current weather conditions from the OpenWeather API.

Next, update the file containing the description of the WeatherFetcher transformer. The contents of this file appears in the Quick Add pane of FME Workbench when the transformer is selected before the package is installed. For instance, it could look like this:

transformers\WeatherFetcher.md
The WeatherFetcher transformer gets the current weather from the OpenWeather API.
An OpenWeather API Key is required.
To get one, visit [openweathermap.org](https://openweathermap.org/).

Finally, update the main documentation for the WeatherFetcher transformer, which appears when the user browses for the transformer's help. This file should describe how to use the transformer, its parameters, and its outputs. For instance, it could look like this:

help\WeatherFetcher.md
# WeatherFetcher (example.weather)

The WeatherFetcher transformer gets the current weather from the OpenWeather API.

## Parameters

### Authorization

* **OpenWeather Connection**: Connection with the OpenWeather API Key.
  To get a key, visit [openweathermap.org](https://openweathermap.org/).

### Coordinates

* **Source**:
 * Parameters: Get weather at the coordinates given by the Latitude and Longitude parameters.
 * Input Geometry: Get weather at the coordinates given by the input geometry.
* **Latitude**: Enabled when Source is set to Parameters.
* **Longitude**: Enabled when Source is set to Parameters.

### Options

* **Units**: Return temperatures in these units.
* **Language**: A language code, like 'fr'. Describe weather in this language.
 Defaults to English if not set.

## Output attributes

* **description**: Text description of the weather.
* **location_name**: Name of the location at the coordinates.
* **temp**: Temperature at the coordinates.

Content shown in Quick Add

The content that is shown in the Quick Add pane of FME Workbench changes depending on whether the package is currently installed. This arrangement allows the Quick Add content to differ between current and prospective users.

When WeatherFetcher is selected but the package is not installed, the text shown is from transformers\WeatherFetcher.md. However, when the package is installed, the text shown is the heading and first paragraph in help\WeatherFetcher.md.

Update package version

Just a few more steps are needed before building a final FPKG file for release and distribution. The package's version number and changelog needs to be updated.

  1. The transformer template starts with a package version of 0.1.0, defined in package.yml. Update version in package.yml to 1.0.0. Remember that package versions must follow Semantic Versioning.
  2. Verify that the version of WeatherFetcher declared in package.yml matches the latest version defined in WeatherFetcher.fmxj.
  3. The transformer's Python module version also needs to be updated. It is set in __init__.py, which is referenced in setup.cfg. Update __version__ in __init__.py to 1.0.0. The version defined here should match the package version, but it is not required.
  4. Add an entry to CHANGES.md to describe what's new. Users will see it in the package's documentation as well as the package's page on FME Hub.
package.yml
fpkg_version: 1
uid: weather
publisher_uid: example
name: My FME Package
description: A short description of my FME Package.
version: 0.1.0
minimum_fme_build: 24158
author:
  name: FME Lizard
package_content:
  transformers:
    - name: WeatherFetcher
      version: 3
  python_packages:
    - name: fme_weatherfetcher
  web_services:
    - name: OpenWeather.xml
python\fme_weatherfetcher\src\fme_weatherfetcher\__init__.py
__version__ = "0.1.0"
CHANGES.md
# My FME Package (example.weather) changes

## 0.1.0

* Initial version.
package.yml
fpkg_version: 1
uid: weather
publisher_uid: example
name: My FME Package
description: A short description of my FME Package.
version: 1.0.0
minimum_fme_build: 24158
author:
  name: FME Lizard
package_content:
  transformers:
    - name: WeatherFetcher
      version: 3
  python_packages:
    - name: fme_weatherfetcher
  web_services:
    - name: OpenWeather.xml
python\fme_weatherfetcher\src\fme_weatherfetcher\__init__.py
__version__ = "1.0.0"
CHANGES.md
# My FME Package (example.weather) changes

## 1.0.0

* Finished implementation of WeatherFetcher.

Make a release

Congratulations! The package is now complete. Run fme-packager pack . to create dist\example.weather-1.0.0.fpkg.

You can now share the package with others and upload it to FME Hub. Refer to the Hello World guide to learn more about FME Hub.