HTTPCaller
Accesses a URL via HTTP or HTTPS, using a variety of HTTP methods.
Available methods include:
HEAD
GET
PUT
POST
DELETE
OPTIONS
PATCH
COPY
LOCK
MKCOL
MOVE
PROPFIND
PROPPATCH
UNLOCK
CHECKOUT
CHECKIN
UNCHECKOUT
LINK
UNLINK
Configuration
Input Ports
This transformer accepts any feature.
Output Ports
The HTTP response body may be saved to an attribute, or a file. By default the body is saved to the attribute identified by the Response Body Attribute parameter.
The HTTP response status code will be stored in the attribute named by the Status Code Attribute parameter. By default this is the _http_status_code attribute. For more information on HTTP Status Codes, please refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.
In addition, the HTTP response headers will be saved in the list attribute specified by the List Attribute for Response Headers parameter. By default the headers are stored in the _headers{} list attribute. For a list of common HTTP response headers, please refer to http://en.wikipedia.org/wiki/List_of_HTTP_header_fields.
If an error occurs while accessing the URL, the feature will be output via the <Rejected> port. In addition, an error message will be set in the attribute named by the Error Attribute parameter.
Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.
Parameters
Request URL |
This parameter identifies the URL which will be accessed by the transformer. The transformer can access http and https URLs. The URL can be entered directly in the transformer, or be built up from attribute values. |
||||||||||
HTTP Method |
Select an HTTP method to access the server. Most requests will use the GET method. Requests that upload data will typically use the PUT, POST or PATCH method, although an upload body can also be provided with the DELETE method. The HEAD, DELETE, and OPTIONS methods can be useful when accessing a REST API. Additional methods are available. |
||||||||||
Query String Parameters |
This table can be used to create additional URL query parameters. The parameters will be appended to the URL given in the Request URL parameter. This can be useful when accessing a REST API endpoint which requires many parameters. The parameter names and values can be entered directly or constructed from attribute values. |
||||||||||
Headers |
This table can be used to create custom HTTP headers which will be sent along with the request. This can be used to customize the request that FME will make to the server. Header names and values can be entered directly or constructed from attribute values. In addition, the Name column provides a list of commonly used header names. |
||||||||||
Body |
These parameters are only enabled when HTTP Method is PUT, POST, DELETE, or PATCH. An upload body is not required; for example, DELETE requests often have no upload body.
|
Save Response Body To |
This parameter allows users to set the destination for the HTTP response body. The body may be saved to an attribute, to a file specified by the user, or to a new file in a directory specified by the user. Note: If you are downloading a very large response body with the HTTPCaller, it is recommended that you save it to a file. This avoids creating a large attribute on the feature. |
||||||
Save Response Body To Attribute |
When this option is selected, each HTTP response body will be saved to an attribute of the feature which made the request. The following parameters are enabled for this option.
|
||||||
Save Response Body To File |
When this option is selected, each HTTP response will be saved to a file. The following parameters are enabled for this option.
|
||||||
Save Response Body To Folder |
When this option is selected, the HTTPCaller will create a new file for each feature that is processed.
|
||||||
Response Header and Status Attributes |
|
||||||
Multipart Response Handling |
It is rare, but possible for an HTTP server to return a multipart response. That is, a single HTTP request will result in an HTTP response which contains several different parts. A multipart response will have a Content-Type header which begins with "multipart", such as "multipart/mixed". The HTTPCaller has the option to split such responses into multiple features. Because multipart responses are very rare, the default behaviour is to not split multipart responses into multiple features. These parameters are only enabled when response bodies are being saved to attributes, or to a folder.
|
Maximum Number of Concurrent HTTP Requests |
This parameter controls how features are processed through the transformer. The value of this parameter indicates the maximum number of transfers that will be in progress at the same time. When the parameter is set to 1, a transfer will be fully processed before the next transfer can begin. When the parameter is set to a value greater than 1, the transformer will not wait for each transfer to finish before starting the next one. There may be multiple transfers occurring at the same time. This can provide significant performance benefits. This value should be adjusted based on the ability of the HTTP server to handle multiple requests at the same time, taking into consideration that multiple HTTPCaller transformers may be accessing the server at the same time. This parameter must be set to 1 if the transformer is being used inside a custom transformer which has a loop output. |
Maximum Requests per Interval |
This is the maximum number of HTTP requests the transformer will start in a particular time interval. For example, if you wanted to limit the transformer to 50 requests per minute, you would set this parameter to 50. |
Interval Length (seconds) |
This is the time period over which requests will be limited. For example, if you wanted to limit the transformer to 50 requests per minute, you would set this parameter to 60. Some preset values are provided in the parameter list, or any number may be manually entered. |
Request Timing |
This controls the way that transfers are started within the time interval. There are two options:
|
Verify SSL Certificates |
If Yes, the transformer will verify SSL certificates. This is done in two stages:
If No, no verification will be performed. This could allow the HTTP request/response to be intercepted without the client or server being aware. On MacOS, requests to some servers will not work unless this parameter is set to Yes. |
Minimum Encryption Strength |
This parameter controls the minimum encryption strength used for HTTPS connections. The options are:
|
Authentication Method |
Select a method for performing authentication. Choices include:
Note: Note that although the HTTP Basic access authentication is a mechanism designed to allow a client to provide credentials to a server on the assumption that the connection between them is trusted and secure, be aware that any credentials passed from client to server can be easily intercepted through an insecure connection. |
Web Connection |
Select or create an FME named Web Connection. |
HTTP Authentication Username |
Provide a username for HTTP authentication methods. |
HTTP Authentication Password |
Provide a password for HTTP authentication methods. |
When enabled, the HTTPCaller will attempt to retry requests which have failed, or have returned an HTTP error.
Error Types to Retry | This parameter allows users to select the types of errors which should lead to a retry attempt. The available options include connection errors such as a network timeout or dns failure, or various HTTP error codes in the 4xx-5xx range. |
Maximum Retry Attempts | This parameter specifies the maximum number of retry attempts that will be made for a single feature, before that feature is output through the <Rejected> port of the transformer. |
Initial Backoff Time (msec) |
This parameter specifies the amount of time that the transformer will wait before retrying a failed request. This value is provided in milliseconds. Subsequent retries of the same request will use a longer backoff time, with the time doubling for each request. This parameter will be ignored if the response to a failed request contains a "Retry-After" HTTP header. If this header is present, the request will be retried after the time or interval specified in the header value. |
Editing Transformer Parameters
Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.
Defining Values
There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.
Using the Text Editor
The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.
Using the Arithmetic Editor
The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.
Conditional Values
Set values depending on one or more test conditions that either pass or fail.
Parameter Condition Definition Dialog
Content
Expressions and strings can include a number of functions, characters, parameters, and more.
When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.
These functions manipulate and format strings. | |
Special Characters |
A set of control characters is available in the Text Editor. |
Math functions are available in both editors. | |
Date/Time Functions | Date and time functions are available in the Text Editor. |
These operators are available in the Arithmetic Editor. | |
These return primarily feature-specific values. | |
FME and workspace-specific parameters may be used. | |
Creating and Modifying User Parameters | Create your own editable parameters. |
Dialog Options - Tables
Transformers with table-style parameters have additional tools for populating and manipulating values.
Row Reordering
|
Enabled once you have clicked on a row item. Choices include:
|
Cut, Copy, and Paste
|
Enabled once you have clicked on a row item. Choices include:
Cut, copy, and paste may be used within a transformer, or between transformers. |
Filter
|
Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output. |
Import
|
Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers. |
Reset/Refresh
|
Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers. |
Note: Not all tools are available in all transformers.
This transformer replaces the HTTPFetcher, HTTPUploader, HTTPFileUploader, HTTPMultipartUploader, HTTPDeleter and HTTPHeader transformers.
FME Community
The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.
Search for samples and information about this transformer on the FME Community.
Keywords: URLFetcher