Reading from a URL
Another variation on a dataset location is the ability to read data from a URL.
Defining a URL dataset to be read is as simple as entering the URL into the Dataset field within any of FME’s dataset selection tools. After reading, the data will be downloaded and used just as if it were a normal dataset.
To add a reader using a URL instead of a file location:
- Open the Add Reader dialog.
- Select the Format.
- Type or paste the URL of a dataset into the Dataset field
- Click OK to add the reader to the workspace.
Reading from Cloud Storage URLs
FME can also read data from a zip file stored in Google
Docs (or from other cloud storage facilities). For example, to read from a Google Doc:
- Share the
file publicly and copy its ID number.
- Open the Add Reader dialog.
- In the Dataset Parameter, use a URL of the form:
https://docs.google.com/uc?export=download&id=XXXX
where XXXX is the ID of the Google document
Usage Notes
- URL datasets can be read using either HTTP or FTP. However, password-protected sites are not currently supported.
- FME can read a dataset that is both a URL and a zip file. However, it is not necessary for all URL datasets to be zipped; non-zipped file datasets can also be read.
- FME cannot currently write data directly to a dataset defined by a URL. In this scenario, the data must be written out to the local file system, and then uploaded to the required location using a shutdown script or an HTTPFileUploader transformer.