Skip to content

Including third-party libraries

If your FME Package needs additional Python libraries, they should be vendorized into the package. Vendorizing a library makes it available to just the code in your package, and avoids conflicts with other instances of the library that may be present in FME or another installed FME Package.

Vendorization is important because FME Packages should be self-contained and include all its dependencies. When FME installs a package, it will not download any dependencies from the Python Package Index (PyPI).

In the examples that follow, vendorization is done using the python-vendorize tool. If you followed the Hello World guide, it should already be installed. Otherwise, install it with pip install vendorize.

Warning

Not all libraries can be successfully vendorized. Whether a library can be vendorized depends on how it is implemented. After vendorizing a library, test the package to ensure that it works as expected.

Warning

Do not vendorize libraries with C extensions. Python libraries with C extensions cannot be fully isolated in FME, which may lead to conflicts and crashes. Binaries also vary between platforms, architectures, and Python versions, which increases complexity.

Example: vendorizing fmetools

The Hello World guide uses a template that involves vendorizing the fmetools library. This library represents the simplest case of vendorization, as it does not have any dependencies aside from FME.

The python-vendorize tool is configured by vendorize.toml. In the package created in the Hello World guide, vendorize.toml looks like this:

python\fme_demogreeter\vendorize.toml
target = "src/fme_demogreeter/_vendor"
packages = [
    "fmetools>=0.7.3",
]

The python-vendorize command reads the configuration in vendorize.toml and places the vendorized version of fmetools into python\fme_demogreeter\src\fme_demogreeter\_vendor. All the package's code under python\fme_demogreeter\src\fme_demogreeter then imports fmetools using the name ._vendor.fmetools.

Libraries included with FME

FME includes a set of Python libraries that are often dependencies of other libraries. Libraries included with FME should not be vendorized into an FME Package.

The Python libraries included with FME are located in FME_HOME\python\ and FME_HOME\python\python3x. Check these folders in the version of FME targeted by your package to see if they include compatible versions of the libraries you need. Popular libraries include Requests, urllib3, and Beautiful Soup.

Example: vendorizing qrcode

Vendorizing libraries that have dependencies involves a few more steps. In this example, we will vendorize the qrcode library to show how to:

  • Identify and vendorize dependencies
  • Identify dependencies that may be omitted
  • Omit vendorized files that are not needed in an FME Package
  • Identify dependencies that are included in FME and should not be vendorized

Vendorize a library and its dependencies

First, we need to check if the library has dependencies. This is important because python-vendorize will not automatically vendorize dependencies, so dependencies must be manually identified and included in vendorize.toml.

The qrcode library's setup.cfg declares three dependencies:

python-qrcode\setup.cfg
[options]
zip_safe = False
include_package_data = True
packages = find:
install_requires =
    colorama;platform_system=="Windows"
    typing_extensions
    pypng
python_requires = >= 3.7

The dependencies that need to be vendorized are typing_extensions and pypng. colorama can be omitted: it's only needed on Windows, and upon closer inspection, it's only used for the qrcode library's command line interface, which is not needed in an FME Package. Therefore, vendorize.toml can be updated to include qrcode and the dependencies:

vendorize.toml
target = "src/fme_example/_vendor"
packages = [
    "fmetools>=0.7.3",
    "qrcode==7.4.2",
    "typing-extensions==4.8.0",
    "pypng==0.20220715.0",
]

typing_extensions and pypng do not have any further dependencies, so the above is the complete list of libraries that need to be vendorized into the package to make the qrcode library available.

Next, run python-vendorize while in the same folder as vendorize.toml. All the listed libraries are downloaded and vendorized into the destination folder. The vendorization process rewrites the qrcode library's dependency imports to point to the vendorized versions.

Your Python code can now import the library from under _vendor:

from ._vendor import qrcode
from ._vendor.qrcode.image import pure
from ._vendor.fmetools.plugins import FMEEnhancedTransformer
from ._vendor.fmetools.paramparsing import TransformerParameterParser

Tip

Vendorize libraries at the beginning of the development process, and ensure that your code imports the library from under _vendor. Do not install an unvendorized version of the library in your development environment. This helps ensure that your code is not accidentally using the unvendorized version, and helps keep your environment consistent between development and deployment in FME.

Vendorized files that may be omitted

After running python-vendorize in the previous step, the _vendor folder has some contents that can optionally be omitted, as they are not needed in an FME Package:

  • bin: This folder contains the command line interface of the qrcode library, which is not applicable in the context of an FME Package.
  • share: This contains documentation for the qrcode command line interface.
  • *.dist-info: These folders contain the metadata for the vendorized libraries.

Omit dependencies included with FME

The qrcode library optionally depends on Pillow. FME includes Pillow, so this optional dependency is already satisfied without needing to vendorize it.