FME Form: 2024.2

Reading from a URL

Another variation on a dataset location is the ability to read data from a URL.

Defining a URL dataset to be read is as simple as entering the URL into the Dataset field within any of FME’s dataset selection tools. After reading, the data will be downloaded and used just as if it were a normal dataset.

URL datasets can be read using either HTTP or FTP.

Note  FME cannot currently write data directly to a dataset defined by a URL. In this scenario, the data must be written out to the local file system, and then uploaded to the required location using a shutdown script or an HTTPFileUploader transformer.

To add a reader using a URL instead of a file location:

  1. Open the Add Reader dialog.
  2. Select the Format.
  3. Type or paste the URL of a dataset into the Dataset field

Note  In the Reader parameters dialog for most formats, the parameter Use Network Authentication is visible when the dataset is a URL.

  1. Click OK to add the reader to the workspace.

Reading from Archive Files

  • FME can read a dataset that is both a URL and an archive or compression-only file.

    • Supported archiving files: 7z, 7zip, rar (except v5), rvz, tar, tar.bz2, tar.gz, tgz, zip

    Supported compression-only files: bz2, gz

    • To read all files of a specified extension at the top level of an archive file, use [*.<extension>]. For example:

      • http://<hostname>/<filename>.zip[*.shp]

    • To read all files of a specified extension at the top level and in folders beneath the top level, use [**\*.<extension>]. For example:

      • http://<hostname>/<filename>.zip[*.shp][**\*.shp]

    • To read all files of a specified extension that are nested two folders deep into an archive file:

      • http://<hostname>/<filename>.zip[*/*/*.shp]

    • To read all files of a specified extension under a relative path, use [<relative\path>\*.<extension>]. For example:

      • http://<hostname>/<filename>.zip[*.shp][canada\lakes\*.shp]

    • If the file extensions inside an archive are different (for example, png and PNG), use *.* to open all files in the specified path. For example:

      • http://<hostname>/<filename>.zip[*.*]

    Reading from Cloud Storage URLs

    FME can also read data from an archive file stored in Google Docs (or from other cloud storage facilities). For example, to read from a Google Doc:

    1. Share the file publicly and copy its ID number.
    2. Open the Add Reader dialog.
    3. In the Dataset Parameter, use a URL of the form:

    https://docs.google.com/uc?export=download&id=XXXX

    where XXXX is the ID of the Google document