HTTPCaller
Accesses a URL via HTTP or HTTPS, performing a GET, PUT, POST, DELETE, HEAD, PATCH or OPTIONS operation.
This transformer replaces the HTTPFetcher, HTTPUploader, HTTPFileUploader, HTTPMultipartUploader, HTTPDeleter and HTTPHeader transformers.
Output
The HTTP response body may be saved to an attribute, or a file. By default the body is saved to the attribute identified by the Response Body Attribute parameter.
The HTTP response status code will be stored in the attribute named by the Status Code Attribute parameter. By default this is the _http_status_code attribute. For more information on HTTP Status Codes, please refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.
In addition, the HTTP response headers will be saved in the list attribute specified by the List Attribute for Response Headers parameter. By default the headers are stored in the _headers{} list attribute. For a list of common HTTP response headers, please refer to http://en.wikipedia.org/wiki/List_of_HTTP_header_fields.
If an error occurs while accessing the URL, the feature will be output via the <Rejected> port. In addition, an error message will be set in the attribute named by the Error Attribute parameter.
Parameters
Request
The HTTP method to access the server. Most requests will use the GET method. Requests that upload data will typically use the PUT, POST or PATCH method, although an upload body can also be provided with the DELETE method. The HEAD, DELETE, and OPTIONS methods can be useful when accessing a REST API.
This parameter identifies the URL which will be accessed by the transformer. The transformer can access http and https URLs. The URL can be entered directly in the transformer, or be built up from attribute values.
Query String Parameters
This table can be used to create additional URL query parameters. The parameters will be appended to the URL given in the Request URL parameter. This can be useful when accessing a REST API endpoint which requires many parameters. The parameter names and values can be entered directly or constructed from attribute values.
Headers
This table can be used to create custom HTTP headers which will be sent along with the request. This can be used to customize the request that FME will make to the server. Header names and values can be entered directly or constructed from attribute values. In addition, the Name column provides a list of commonly used header names.
Body
These parameters are only enabled when HTTP Method is PUT, POST, DELETE, or PATCH. An upload body is not required; for example, DELETE requests often have no upload body.
This parameter identifies the type of data that will be uploaded. The transformer can upload a file, a string value, or a multi-part combination of file and string values.
Upload data may be entered using this parameter. The data may be entered directly into the transformer or constructed from attribute values.
When uploading a file, this parameter provides the file path. The path may be entered directly into the transformer or set from an attribute value.
When uploading a string value or a file, this parameter provides the data content type. The value of this parameter will be sent in the Content-Type request header. The drop-down list contains a number of commonly-used content type values. If a different value is required, it can be typed into the text field, or derived from an attribute value.
This table is used to construct multipart uploads. Each part in a multipart upload must have a name, which is given in the first column. The second column identifies the type of part – either a file or a string value. The third column provides the data and MIME type for the part.
Response
This parameter allows users to set the destination for the HTTP response body. The body may be saved to an attribute, or to a file.
Note: If you are downloading a very large response body with the HTTPCaller, it is recommended that you save it to a file. This avoids creating a large attribute on the feature.
When saving the response body to an attribute, this parameter names the attribute which will be used. The default attribute name is _response_body. When performing a HTTP HEAD request, this attribute will be empty.
This parameter is used to indicate the character encoding of the target attribute. By default, the attribute encoding will be automatically determined from the HTTP response headers. Alternatively, a specific encoding may be selected from the list. When downloading binary data, the parameter should be set to Binary or Auto Detect from HTTP Headers.
Save Response Body To File
This parameter controls the manner in which the transformer will save response bodies to files. There are two modes:
- No: The response body will be saved to the file specified in the Output Filename parameter.
- Yes: A new file will be created for each feature, and saved in the folder specified in the Output Folder parameter.
When the Create a New File Per Feature parameter is set to No, the transformer will save HTTP response bodies to the file specified in this parameter. The file may be selected directly, or set from an attribute. Note that if the file is selected directly, the file will be overwritten by each feature passing through the transformer. If more than one feature is passing through the transformer, it is recommended that the file path be set to a unique attribute value, or that the transformer be configured to create a new file for each feature.
When the Create a New File Per Feature parameter is set to Yes, the transformer will create a new file for each feature passing through the transformer. The files will be saved in the folder specified in this parameter.
If Create a New File Per Feature is Yes, this parameter specifies the file type extension for the new file names. Select from a list of common file types, type in manually, or automatically set based on the Content-Type header in the HTTP response.
After the response body has been saved to a file, the attribute named in this parameter will be set to the file path.
Response Headers and Status
This parameter is optional. If it is set to a list attribute name, then HTTP headers returned by the server will be stored in the list attribute. The headers will be split into names and values. For example:
`_headers{0}.name' has value `X-XSS-Protection'
`_headers{0}.value' has value `1; mode=block'
`_headers{1}.name' has value `Date'
`_headers{1}.value' has value `Wed, 25 May 2011 17:20:20 GMT'
`_headers{2}.name' has value `Transfer-Encoding'
`_headers{2}.value' has value `chunked'
`_headers{3}.name' has value `Content-Type'
`_headers{3}.value' has value `text/html; charset=ISO-8859-1'
`_headers{4}.name' has value `Cache-Control'
`_headers{4}.value' has value `private, max-age=0'
This parameter is optional. If it is set to an attribute name, then the HTTP response code returned by the server will be stored in the attribute.
This parameter is optional. If an error occurs, the feature will be output through the <Rejected> output port, and the attribute named in this parameter will contain a brief error message.
HTTP Client Options
If this parameter is set to Yes, any saved cookies that correspond to the URL being accessed will be sent with the request. In addition, any cookies contained in the response will be saved for use with subsequent HTTP requests.
Saved cookies will persist for the duration of the translation – they cannot be used in subsequent translations.
- An HTTPCaller transformer is used to log in to the web service and a session cookie is saved.
- A second HTTPCaller transformer is used to access the web service, utilizing the saved session cookie.
- A third HTTPCaller is used to log out of the web service, which invalidates the saved cookie.
If this parameter is set to Yes, the transformer will follow any URL redirects it encounters, making a second request to a new location.
If this parameter is set to Yes, the transformer will verify SSL certificates. This is done in two stages:
- Ensure that the certificate is for the host we are trying to connect to.
- Ensure that the certificate is valid.
This parameter can be used to set a timeout on HTTP connection requests. If a connection to the server cannot be created within the given number of seconds, the request will fail, and the feature will be output through the <Rejected> port. The default connection timeout is 60 seconds. If 0 is entered, there will be no timeout, and the transformer will wait indefinitely for a successful connection.
This parameter can be used to set a timeout on HTTP transfers. If data stops being transferred for more than the given number of seconds, the request will fail, and the feature will be output through the <Rejected> port. The default transfer timeout is 90 seconds. If 0 is entered, there will be no timeout, and the transformer will wait indefinitely for data to resume being transferred.
Authentication Parameters
The transformer supports user authentication in two ways: Named Connections and username/password authentication.
The Named Connection option allows the selection of a stored Named Connection, which will authenticate the user with the appropriate Web Service. To add a new Named Connection, select “Add Connection…” from the Named Connection dropdown menu. The Named Connection option allows for OAuth2 authentication with supported Web Services.
When authenticating with a username and password, the basic, digest and NTLM access authentication methods are supported:
- Basic (HTTP Basic authentication): This is the default choice, and the only method in widespread use and supported virtually everywhere. This sends the username and password over the network in plain text, easily captured by others.
- Digest (HTTP Digest authentication): Digest authentication is more secure over public networks than the Basic method.
- NTLM (HTTP NTLM authentication): NTLM is a proprietary protocol invented and used by Microsoft. It uses a challenge-response and hash concept similar to Digest to prevent the password from being eavesdropped.
Note: Note that although the HTTP Basic access authentication is a mechanism designed to allow a client to provide credentials to a server on the assumption that the connection between them is trusted and secure, be aware that any credentials passed from client to server can be easily intercepted through an insecure connection.
Editing Transformer Parameters
Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.
Defining Values
There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.
Using the Text Editor
The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.
Using the Arithmetic Editor
The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.
Conditional Values
Set values depending on one or more test conditions that either pass or fail.
Parameter Condition Definition Dialog
Content
Expressions and strings can include a number of functions, characters, parameters, and more.
When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.
These functions manipulate and format strings. | |
Special Characters |
A set of control characters is available in the Text Editor. |
Math functions are available in both editors. | |
Date/Time Functions | Date and time functions are available in the Text Editor. |
These operators are available in the Arithmetic Editor. | |
These return primarily feature-specific values. | |
FME and workspace-specific parameters may be used. | |
Creating and Modifying User Parameters | Create your own editable parameters. |
Dialog Options - Tables
Transformers with table-style parameters have additional tools for populating and manipulating values.
Row Reordering
|
Enabled once you have clicked on a row item. Choices include:
|
Cut, Copy, and Paste
|
Enabled once you have clicked on a row item. Choices include:
Cut, copy, and paste may be used within a transformer, or between transformers. |
Filter
|
Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output. |
Import
|
Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers. |
Reset/Refresh
|
Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers. |
Note: Not all tools are available in all transformers.
FME Licensing Level
FME Professional edition and above
FME Community
The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.
Search for samples and information about this transformer on the FME Community.
Keywords: URLFetcher