HTTPCaller

Categories

Web

Workflows

Accesses a URL via HTTP or HTTPS, using a variety of HTTP methods.

Available methods include:

HEAD
GET
PUT
POST
DELETE
OPTIONS
PATCH
COPY
LOCK
MKCOL
MOVE
PROPFIND
PROPPATCH
UNLOCK
CHECKOUT
CHECKIN
UNCHECKOUT
LINK
UNLINK

Import from OpenAPI Specification

Requests may be imported from APIs that provide an OpenAPI specification JSON or YAML file. To do this:

  • Download the specification file

  • Use the Import > Import from OpenAPI Specification button to access the import dialog

  • Provide the file location in the Specification parameter

  • Select from the available endpoints, and the HTTPCaller dialog will be populated as specified, including Request URL, Authentication Method, and request details such as HTTP Method and Query String Parameters.

Note that any selections made here will overwrite current parameter values.

Examples

This example uses an OpenAPI example specification to populate the HTTPCaller.

The example JSON file has been downloaded from OpenAPI.

Use the Import button, Import from OpenAPI Specification.

After providing the file location, the spec is loaded and available Endpoints are listed.

Our selection provides a Request URL, sets HTTP Method to GET, and populates some Query String Parameters. Authentication was not required.

 

Configuration

Input Ports

This transformer accepts any feature.

Output Ports

The HTTP response body may be saved to an attribute, or a file. By default the body is saved to the attribute identified by the Response Body Attribute parameter.

The HTTP response status code will be stored in the attribute named by the Status Code Attribute parameter. By default this is the _http_status_code attribute. For more information on HTTP Status Codes, please refer to https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

In addition, the HTTP response headers will be saved in the list attribute specified by the Multipart Header List Attribute parameter. By default the headers are stored in the _headers{} list attribute. For a list of common HTTP response headers, please refer to https://en.wikipedia.org/wiki/List_of_HTTP_header_fields.

If an error occurs while accessing the URL, the feature will be output via the <Rejected> port. In addition, an error message will be set in the attribute named by the Error Attribute parameter.

Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.

Parameters

Request URL

This parameter identifies the URL which will be accessed by the transformer. The transformer can access http and https URLs. The URL can be entered directly in the transformer, or be built up from attribute values.

HTTP Method

Select an HTTP method to access the server.

Most requests will use the GET method. Requests that upload data will typically use the PUT, POST or PATCH method, although an upload body can also be provided with the DELETE method. The HEAD, DELETE, and OPTIONS methods can be useful when accessing a REST API.

Additional methods are available.

Use Authentication

If enabled, provide authentication details as needed.

Authentication Method

Select a method for performing authentication. Choices include: 

  • Basic: (HTTP Basic authentication): This is the default choice, and the only method in widespread use and supported virtually everywhere. This sends the username and password over the network in plain text, easily captured by others.
  • Digest: (HTTP Digest authentication): Digest authentication is more secure over public networks than the Basic method.
  • NTLM: (HTTP NTLM authentication): NTLM is a proprietary protocol invented and used by Microsoft. It uses a challenge-response and hash concept similar to Digest to prevent the password from being eavesdropped.
  • Web Connection: Used a named Web Connection, allowing OAuth2 authentication with supported Web Services. New connections may be defined from the Web Connection parameter drop-down.
  • Single Sign-On: When running on Windows, Single Sign-On allows for the use of NTLM authentication using the credentials of the current Windows user account. This is more secure as it allows users to avoid saving their credentials in the transformer parameters. It also allows separate users to run the workspace without having to modify the transformer parameters to contain their own credentials. This parameter has no effect on the Linux or MacOS platforms.

Note:  Note that although the HTTP Basic access authentication is a mechanism designed to allow a client to provide credentials to a server on the assumption that the connection between them is trusted and secure, be aware that any credentials passed from client to server can be easily intercepted through an insecure connection.  

Web Connection

Select or create an FME named Web Connection.

HTTP Authentication Username

Provide a username for HTTP authentication methods.

HTTP Authentication Password

Provide a password for HTTP authentication methods.

Query String Parameters

This table can be used to create additional URL query parameters. The parameters will be appended to the URL given in the Request URL parameter. This can be useful when accessing a REST API endpoint which requires many parameters. The parameter names and values can be entered directly or constructed from attribute values.

Headers

This table can be used to create custom HTTP headers which will be sent along with the request. This can be used to customize the request that FME will make to the server. Header names and values can be entered directly or constructed from attribute values. In addition, the Name column provides a list of commonly used header names.

Body

These parameters are only enabled when HTTP Method is PUT, POST, DELETE, or PATCH. An upload body is not required; for example, DELETE requests often have no upload body.

Upload Data

This parameter identifies the type of data that will be uploaded. The transformer can upload a file, a string value, or a multi-part combination of file and string values.

Upload Body

Upload data may be entered using this parameter. The data may be entered directly into the transformer or constructed from attribute values.

Upload File

When uploading a file, this parameter provides the file path. The path may be entered directly into the transformer or set from an attribute value.

Content Type

When uploading a string value or a file, this parameter provides the data content type. The value of this parameter will be sent in the Content-Type request header. The drop-down list contains a number of commonly-used content type values. If a different value is required, it can be typed into the text field, or derived from an attribute value.

Multipart Upload

This table is used to construct multipart uploads.

Name

Each part must have a name.

Upload Type

Select an upload type:

  • File Upload

  • String Upload

Value

Provide either a string or file for upload, per the Type selection.

A MIME Type will be assigned, and may also be manually set.

When Type is File Upload, the default MIME Type is application/octet-stream. When Type is String Upload, the default is text/plain.

MIME Type and data for upload (String or File) may be accessed via the ellipsis button.

Save Response Body To

This parameter allows users to set the destination for the HTTP response body. The body may be saved to an attribute, to a file specified by the user, or to a new file in a directory specified by the user.

Note: If you are downloading a very large response body with the HTTPCaller, it is recommended that you save it to a file. This avoids creating a large attribute on the feature.

Save Response Body To Attribute

When this option is selected, each HTTP response body will be saved to an attribute of the feature which made the request. The following parameters are enabled for this option.

Response Body Attribute

This parameter names the attribute which will be used to store the response body. The default attribute name is _response_body. When performing a HTTP HEAD request, this attribute will be empty. This attribute may also be empty if the HTTP response does not have a body, for example with an HTTP 304 response.

Response Body Encoding

This parameter is used to indicate the character encoding of the target attribute. By default, the attribute encoding will be automatically determined from the HTTP response headers. Alternatively, a specific encoding may be selected from the list. When downloading binary data, the parameter should be set to Binary or Auto Detect from HTTP Headers.

Save Response Body To File

When this option is selected, each HTTP response will be saved to a file. The following parameters are enabled for this option.

Output Filename

This parameter specifies the file which will be used when saving response bodies to a file. The file may be selected directly, or set from an attribute. Note that if the file is selected directly, the file will be overwritten by each feature passing through the transformer. If more than one feature is passing through the transformer, it is recommended that the file path be set to a unique attribute value, or that the transformer be configured to download to a directory.

File Path Attribute

After the response body has been saved to a file, the attribute named in this parameter will be set to the file path. This allows the file to be manipulated by other transformers in the workspace.

Save Response Body To Folder

When this option is selected, the HTTPCaller will create a new file for each feature that is processed.

Output Folder

This parameter specifies the folder to which response body files will be saved.

File Extension

This parameter specifies the file type extension for the new file names. The file extension can be selected from a list of common file types, typed in manually, or automatically set based on the Content-Type header in the HTTP response.

File Path Attribute

After the response body has been saved to a file, the attribute named in this parameter will be set to the file path.

Response Header and Status Attributes

Header List Attribute

This parameter is optional. If it is set to a list attribute name, then HTTP headers returned by the server will be stored in the list attribute. The headers will be split into names and values. For example:

`_headers{0}.name' has value `X-XSS-Protection'

`_headers{0}.value' has value `1; mode=block'

`_headers{1}.name' has value `Date'

`_headers{1}.value' has value `Wed, 25 May 2011 17:20:20 GMT'

`_headers{2}.name' has value `Transfer-Encoding'

`_headers{2}.value' has value `chunked'

`_headers{3}.name' has value `Content-Type'

`_headers{3}.value' has value `text/html; charset=ISO-8859-1'

`_headers{4}.name' has value `Cache-Control'

`_headers{4}.value' has value `private, max-age=0'

Status Code Attribute

This parameter is optional. If it is set to an attribute name, then the HTTP response code returned by the server will be stored in the attribute.

Error Attribute

This parameter is optional. If an error occurs, the feature will be output through the <Rejected> output port, and the attribute named in this parameter will contain a brief error message.

Multipart Response Handling

It is rare, but possible for an HTTP server to return a multipart response. That is, a single HTTP request will result in an HTTP response which contains several different parts. A multipart response will have a Content-Type header which begins with "multipart", such as "multipart/mixed". The HTTPCaller has the Feature Output option to split such responses into multiple features. Because multipart responses are very rare, the default behavior is to not split multipart responses into multiple features.

These parameters are only enabled when response bodies are being saved to attributes, or to a folder.

Feature Output

This parameter controls the transformer behavior when a multipart response is received.

Single Feature (default): One feature will be output containing the entire response. The number of features output from the transformer will equal the number of features which enter the transformer.

Multiple Features: A separate feature will be output for each part of a multipart response. There may be more output features than input features. If response bodies are being saved to a folder, a subfolder will be created for the multipart response, and each part will be saved to a file within this subfolder.

Multipart Header List Attribute

Each part in a multipart response may have its own set of headers. When multipart responses are split into separate features, each feature will have a list attribute containing the names and values of the part's headers.

This parameter specifies the name of that list attribute. This list will be populated in addition to the list attribute specified by the Header List Attribute parameter, and these two parameters should have different values.

Concurrent Requests

Maximum Number of Concurrent HTTP Requests

This parameter controls how features are processed through the transformer. The value of this parameter indicates the maximum number of transfers that will be in progress at the same time.

When the parameter is set to 1, a transfer will be fully processed before the next transfer can begin.

When the parameter is set to a value greater than 1, the transformer will not wait for each transfer to finish before starting the next one. There may be multiple transfers occurring at the same time. This can provide significant performance benefits. This value should be adjusted based on the ability of the HTTP server to handle multiple requests at the same time, taking into consideration that multiple HTTPCaller transformers may be accessing the server at the same time.

This parameter must be set to 1 if the transformer is being used inside a custom transformer which has a loop output.

Retry Failed Requests

When enabled, the HTTPCaller will attempt to retry requests which have failed, or have returned an HTTP error.

Error Types to Retry

This parameter allows users to select the types of errors which should lead to a retry attempt. The available options include connection errors such as a network timeout or dns failure, or various HTTP error codes in the 4xx-5xx range.

Maximum Retry Attempts

This parameter specifies the maximum number of retry attempts that will be made for a single feature, before that feature is output through the <Rejected> port of the transformer.

Initial Backoff Time (msec)

This parameter specifies the amount of time that the transformer will wait before retrying a failed request. This value is provided in milliseconds. Subsequent retries of the same request will use a longer backoff time, with the time doubling for each request.

This parameter will be ignored if the response to a failed request contains a "Retry-After" HTTP header. If this header is present, the request will be retried after the time or interval specified in the header value.

 

Rate Limiting

Maximum Requests per Interval

This is the maximum number of HTTP requests the transformer will start in a particular time interval. For example, if you wanted to limit the transformer to 50 requests per minute, you would set this parameter to 50.

Interval Length (seconds)

This is the time period over which requests will be limited. For example, if you wanted to limit the transformer to 50 requests per minute, you would set this parameter to 60. Some preset values are provided in the parameter list, or any number may be manually entered.

Request Timing

This controls the way that transfers are started within the time interval. There are two options:

  • Start requests as soon as possible: When this value is selected, requests will be started as soon as possible, as long as the maximum number of requests started in the preceding time interval does not exceed the maximum. This may lead to many requests being started at the beginning of the time interval, and then nothing happening until the interval has passed.

  • Start requests evenly through the interval: When this value is selected, the transformer will attempt to start requests evenly through the interval. That is, it will divide the interval length by the maximum number of requests, and then ensure that at least this much time elapses between starting each request. This is equivalent to inserting a Decelerator transformer before the HTTPCaller, and can ease the load on an HTTP server.

HTTP Client Options

HTTP Version

This parameter allows users to control which version of the HTTP protocol is used in the transformer.

When Restrict to HTTP/1.1 is selected, the use of HTTP/2 will be disabled and only HTTP/1.1 will be used.

When Use HTTP/2 if possible is selected, HTTP/2 may be used if the HTTP server supports it. Otherwise HTTP/1.1 will be used.

Typically this parameter should be configured to allow the use of HTTP/2. Restricting the transformer to HTTP/1.1 should only be used when connecting to servers which do not handle HTTP/2 correctly.

Save Cookies

If this parameter is set to Yes, any saved cookies that correspond to the URL being accessed will be sent with the request. In addition, any cookies contained in the response will be saved for use with subsequent HTTP requests.

Saved cookies will persist for the duration of the translation – they cannot be used in subsequent translations.

Saving cookies is useful when a web service keeps a "session". Then the following sequence of operations might be used:

  1. An HTTPCaller transformer is used to log in to the web service and a session cookie is saved.
  2. A second HTTPCaller transformer is used to access the web service, utilizing the saved session cookie.
  3. A third HTTPCaller is used to log out of the web service, which invalidates the saved cookie.

Follow Redirects

This parameter controls how redirects are handled in the HTTPCaller. The options are:

  • Yes - Use GET for all redirect requests : When this option is chosen, the transformer will perform a GET request when it is redirected. If the original request had an upload body, this option prevents the upload data from being included in the redirect request.

  • Yes - Use POST if the original request was a POST: When this option is chosen, the redirect request will use the same method as the original request. This may mean that upload data is included in the redirect request, even when the redirect is to a different domain.

  • No - do not follow redirects: When this option is chosen, redirect responses will not be automatically followed.

Redirect Authentication

This parameter controls how authentication related request headers (if there are any) are handled when following redirects. The options are:

  • Send username and password to the first domain only: In this case, the authentication headers will only be sent to the domain provided in the Request URL parameter. This is the more secure option.

  • Send username and password to all domains: In this case, the authentication headers will be sent in all requests, even when following a redirect to a different domain. This is less secure, as it could lead to a username/password being exposed to servers they are not intended for.

Connection Timeout (seconds)

This parameter can be used to set a timeout on HTTP connection requests. If a connection to the server cannot be created within the given number of seconds, the request will fail, and the feature will be output through the <Rejected> port. The default connection timeout is 60 seconds. If 0 is entered, there will be no timeout, and the transformer will wait indefinitely for a successful connection.

Transfer Timeout (seconds)

This parameter can be used to set a timeout on HTTP transfers. If data stops being transferred for more than the given number of seconds, the request will fail, and the feature will be output through the <Rejected> port. The default transfer timeout is 90 seconds. If 0 is entered, there will be no timeout, and the transformer will wait indefinitely for data to resume being transferred.

Advanced Security

Verify HTTPS Certificates

If Yes, the transformer will verify HTTPS certificates. This is done in two stages:

  1. Ensure that the certificate is for the host we are trying to connect to.
  2. Ensure that the certificate is valid. That is, it has been signed by a trusted CA authority.

If No, no verification will be performed. This could allow the HTTP request/response to be intercepted without the client or server being aware.

On MacOS, requests to some servers will not work unless this parameter is set to Yes.

Minimum Encryption Strength

This parameter controls the minimum encryption strength used for HTTPS connections. The options are:

  • Use Operating System Default Settings: When this option is chosen, connections will be made with the security settings recommended by each platform.

  • Allow Less Secure Encryption Algorithms: When this option is chosen, connections can be made to servers which do not meet the security level recommended by the operating system.

    On Linux, selecting this option corresponds to the "Level 1" security, as documented here.

    On Windows, this corresponds to not using the SCH_USE_STRONG_CRYPTO flag, as documented here.

Specification

Specify a JSON or YAML OpenAPI specification file to use. This file must be downloaded (URLs are not supported).

Select Endpoint

Select an available option to have its details populated in the main parameter dialog.

Editing Transformer Parameters

Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.

Using the Text Editor

The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.

Text Editor

Using the Arithmetic Editor

The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.

Arithmetic Editor

Conditional Values

Set values depending on one or more test conditions that either pass or fail.

Parameter Condition Definition Dialog

Content

Expressions and strings can include a number of functions, characters, parameters, and more.

When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.

String Functions

These functions manipulate and format strings.

Special Characters

 A set of control characters is available in the Text Editor.

Math Functions

Math functions are available in both editors.
Date/Time Functions Date and time functions are available in the Text Editor.

Math Operators

These operators are available in the Arithmetic Editor.

FME Feature Functions

These return primarily feature-specific values.

FME Parameters

FME and workspace-specific parameters may be used.
Creating and Modifying User Parameters Create your own editable parameters.

Dialog Options - Tables

Transformers with table-style parameters have additional tools for populating and manipulating values.

Row Reordering

Enabled once you have clicked on a row item. Choices include:

  • Add a row
  • Remove a row
  • Move current row up one
  • Move current row down one
  • Move current row to top
  • Move current row to bottom

Cut, Copy, and Paste

Enabled once you have clicked on a row item. Choices include:

  • Cut a row - delete and copy to clipboard
  • Copy a row to the clipboard
  • Paste a row from the clipboard

Cut, copy, and paste may be used within a transformer, or between transformers.

Filter

Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output.

Import

Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers.

Reset/Refresh

Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers.

Note: Not all tools are available in all transformers.

This transformer replaces the HTTPFetcher, HTTPUploader, HTTPFileUploader, HTTPMultipartUploader, HTTPDeleter and HTTPHeader transformers.

FME Community

The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.

Search for all results about the HTTPCaller on the FME Community.

Keywords: URLFetcher