FME Transformers: 2025.0
Connects to a Twitter™ stream and outputs a new feature for each tweet. The transformer allows users to set up a filter, so that the output features will only contain tweets containing certain keywords, tweets that are geotagged with a particular location, and/or tweets which are written in particular languages.
A twitter stream is an HTTP connection that never closes, so except for error cases, any workspace containing this transformer will not terminate on its own. The transformer will continue to output features until the workspace is terminated by the user.
At least one feature must enter the transformer in order to start the stream. Once all the input features have arrived, the transformer will open a connection to the Twitter streaming API, and begin outputting features. Depending on the filters set up in the transformer parameters, the output from the transformer may not be continuous. There may be periods in which no tweets match the filters. This is particularly true when filtering by language and geometry.
If no keywords or geometry filters are provided, the transformer will connect to a sample stream. This stream provides a random sample of tweets as they are created. It is possible to filter this stream by language.
Note that a limited number of streaming connections is permitted per user. If too many simultaneous connections are attempted with the same user credentials, one or more of the connections will be terminated.
Configuration
Output Ports
Tweets
The transformer will output a new feature for each tweet received from the twitter streaming API. Each output feature will contain several attributes:
_tweet_id: The identification number of the tweet.
_tweet_status: The text of the tweet.
_tweet_username: The username of the Twitter user who wrote the tweet.
_tweet_created_at: The date and time that the tweet was created. This will be formatted in the standard FME date/time format: YYYYMMDDHHMMSS
_tweet_search_result: The JSON text of the tweet. Further information about the tweet can be extracted from this value using the JSONFragmenter and JSONExtractor transformers.
Other
The Twitter streaming API will periodically return messages which are not tweets. For example, these messages could contain information on rate limiting or deleted messages. More information can be found here: https://dev.twitter.com/docs/streaming-apis/messages
One feature will be output through this port for each non-tweet message returned from the Twitter streaming API. Each of these features will have a _tweet_search_result attribute, which will contain the JSON object representation of the message. This object can be examined in FME Workbench using the JSON transformers.
Parameters
Stream Filter
|
Keywords |
This parameter is used to filter the stream for tweets which contain the given keywords. For each input feature, the value should contain a space separated list of keywords. If a tweet contains all of the keywords from any single feature, it will be returned by the stream. For example, if one feature provides the keywords “FME Workbench” while a second provides “safe”, the following tweets could be matched:
The following tweets would NOT be matched:
Keywords are matched case insensitively. Furthermore, the keywords will be matched if they are contained in the Twitter username, a linked URL or Twitter hashtags. For more details, look at https://dev.twitter.com/docs/streaming-apis/parameters#track |
|
Languages |
This parameter can be used to restrict tweets to the selected languages. If no languages are selected, the streamed tweets could be in any language. |
|
Filter by Feature Geometry |
If this parameter is set to Yes the bounding box of each input features will be calculated, and the tweet stream will be restricted to tweets which are geotagged, and whose coordinates fall within one of the bounding boxes. |
|
Buffer for Point Features |
This parameter only takes effect if the stream is being filtered by geometry. Since the stream requires a bounding box, this parameter determines the size of bounding box which will be used for point geometries. |
Twitter Account
|
Account |
Select or create a Web Connection to connect to a Twitter account. |
Editing Transformer Parameters
Transformer parameters can be set by directly entering values, using expressions, or referencing other elements in the workspace such as attribute values or user parameters. Various editors and context menus are available to assist. To see what is available, click 
beside the applicable parameter.
How to Set Parameter Values
Defining Values
There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters.
Using the Text Editor
The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.
Using the Arithmetic Editor
The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.
Conditional Values
Set values depending on one or more test conditions that either pass or fail.
Parameter Condition Definition Dialog
Content
Expressions and strings can include a number of functions, characters, parameters, and more.
When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.
Content Types
| These functions manipulate and format strings. | |
|
Special Characters |
A set of control characters is available in the Text Editor. |
| Math functions are available in both editors. | |
| Date/Time Functions | Date and time functions are available in the Text Editor. |
| These operators are available in the Arithmetic Editor. | |
| These return primarily feature-specific values. | |
| FME and workspace-specific parameters may be used. | |
| Creating and Modifying User Parameters | Create your own editable parameters. |
Dialog Options - Tables
Table Tools
Transformers with table-style parameters have additional tools for populating and manipulating values.
|
Row Reordering
|
Enabled once you have clicked on a row item. Choices include:
|
|
Cut, Copy, and Paste
|
Enabled once you have clicked on a row item. Choices include:
Cut, copy, and paste may be used within a transformer, or between transformers. |
|
Filter
|
Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output. |
|
Import
|
Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers. |
|
Reset/Refresh
|
Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers. |
Note: Not all tools are available in all transformers.
For more information, see Transformer Parameter Menu Options.
Reference
|
Processing Behavior |
|
|
Feature Holding |
No |
| Dependencies | None |
| Aliases | TwitterStreamer |
| History |
FME Community
The FME Community has a wealth of FME knowledge with over 20,000 active members worldwide. Get help with FME, share knowledge, and connect with users globally.
Search for all results about the TweetStreamer on the FME Community.
Examples may contain information licensed under the Open Government Licence – Vancouver, Open Government Licence - British Columbia, and/or Open Government Licence – Canada.