Reading from a URL

Another variation on a dataset location is the ability to read data from a URL.

Defining a URL dataset to be read is as simple as entering the URL into the Dataset field within any of FME’s dataset selection tools. After reading, the data will be downloaded and used just as if it were a normal dataset.

To add a reader using a URL instead of a file location:

  1. Open the Add Reader dialog.
  2. Select the Format.
  3. Type or paste the URL of a dataset into the Dataset field

Note: In the Reader parameters dialog for most formats, the parameter Use Network Authentication is visible when the dataset is a URL.

  1. Click OK to add the reader to the workspace.

Reading from Cloud Storage URLs

FME can also read data from a zip file stored in Google Docs (or from other cloud storage facilities). For example, to read from a Google Doc:

  1. Share the file publicly and copy its ID number.
  2. Open the Add Reader dialog.
  3. In the Dataset Parameter, use a URL of the form:

https://docs.google.com/uc?export=download&id=XXXX

where XXXX is the ID of the Google document

Usage Notes

  • URL datasets can be read using either HTTP or FTP.
  • FME can read a dataset that is both a URL and a zip file. However, it is not necessary for all URL datasets to be zipped; non-zipped file datasets can also be read.
  • FME cannot currently write data directly to a dataset defined by a URL. In this scenario, the data must be written out to the local file system, and then uploaded to the required location using a shutdown script or an HTTPFileUploader transformer.