Connects to a Twitter™ stream and outputs a new feature for each tweet. The transformer allows users to set up a filter, so that the output features will only contain tweets containing certain keywords, tweets that are geotagged with a particular location, and/or tweets which are written in particular languages.
A twitter stream is an HTTP connection that never closes, so except for error cases, any workspace containing this transformer will not terminate on its own. The transformer will continue to output features until the workspace is terminated by the user.
At least one feature must enter the transformer in order to start the stream. Once all the input features have arrived, the transformer will open a connection to the Twitter streaming API, and begin outputting features. Depending on the filters set up in the transformer parameters, the output from the transformer may not be continuous. There may be periods in which no tweets match the filters. This is particularly true when filtering by language and geometry.
If no keywords or geometry filters are provided, the transformer will connect to a sample stream. This stream provides a random sample of tweets as they are created. It is possible to filter this stream by language.
Note that a limited number of streaming connections is permitted per user. If too many simultaneous connections are attempted with the same user credentials, one or more of the connections will be terminated.
Output Ports
Tweets
The transformer will output a new feature for each tweet received from the twitter streaming API. Each output feature will contain several attributes:
_tweet_id: The identification number of the tweet.
_tweet_status: The text of the tweet.
_tweet_username: The username of the Twitter user who wrote the tweet.
_tweet_created_at: The date and time that the tweet was created. This will be formatted in the standard FME date/time format: YYYYMMDDHHMMSS
_tweet_search_result: The JSON text of the tweet. Further information about the tweet can be extracted from this value using the JSONFragmenter and JSONExtractor transformers.
Other
The Twitter streaming API will periodically return messages which are not tweets. For example, these messages could contain information on rate limiting or deleted messages. More information can be found here: http://dev.twitter.com/docs/streaming-apis/messages
One feature will be output through this port for each non-tweet message returned from the Twitter streaming API. Each of these features will have a _tweet_search_result attribute, which will contain the JSON object representation of the message. This object can be examined in Workbench using the JSON transformers.
Parameters
Stream Filter Parameters
Keywords
This parameter is used to filter the stream for tweets which contain the given keywords. For each input feature, the value should contain a space separated list of keywords. If a tweet contains all of the keywords from any single feature, it will be returned by the stream.
For example, if one feature provides the keywords “FME Workbench” while a second provides “safe”, the following tweets could be matched:
- FME Workbench
- Safe Software
The following tweets would NOT be matched:
- Twitter Workbench
- FME
Keywords are matched case insensitively. Furthermore, the keywords will be matched if they are contained in the Twitter username, a linked URL or Twitter hashtags. For more details, look at http://dev.twitter.com/docs/streaming-apis/parameters#track
Languages
This parameter can be used to restrict tweets to the selected languages. If no languages are selected, the streamed tweets could be in any language.
Filter by Feature Geometry
If this parameter is set to Yes the bounding box of each input features will be calculated, and the tweet stream will be restricted to tweets which are geotagged, and whose coordinates fall within one of the bounding boxes.
Buffer for Point Features
This parameter only takes effect if the stream is being filtered by geometry. Since the stream requires a bounding box, this parameter determines the size of bounding box which will be used for point geometries.
Authentication Parameters
Note: For access outside of your network using a proxy server, use the Network tools in FME Options. From the Workbench menu, select Tools > FME Options > Network. For more information, see "Network" in the FME Workbench Help.
Twitter Username and Password
A valid Twitter username and password are required in order to access the Twitter streaming API.
Editing Transformer Parameters
Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click 
beside the applicable parameter. For more information, see Transformer Parameter Menu Options.
Defining Values
There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.
How to Set Parameter Values
Using the Text Editor
The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.
Using the Arithmetic Editor
The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.
Conditional Values
Set values depending on one or more test conditions that either pass or fail.
Parameter Condition Definition Dialog
Content
Expressions and strings can include a number of functions, characters, parameters, and more.
When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.
Content Types
| These functions manipulate and format strings. | |
|
Special Characters |
A set of control characters is available in the Text Editor. |
| Math functions are available in both editors. | |
| Date/Time Functions | Date and time functions are available in the Text Editor. |
| These operators are available in the Arithmetic Editor. | |
| These return primarily feature-specific values. | |
| FME and workspace-specific parameters may be used. | |
| Creating and Modifying User Parameters | Create your own editable parameters. |
Dialog Options - Tables
Transformers with table-style parameters have additional tools for populating and manipulating values.
Table Tools
|
Row Reordering
|
Enabled once you have clicked on a row item. Choices include:
|
|
Cut, Copy, and Paste
|
Enabled once you have clicked on a row item. Choices include:
Cut, copy, and paste may be used within a transformer, or between transformers. |
|
Filter
|
Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output. |
|
Import
|
Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers. |
|
Reset/Refresh
|
Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers. |
Note: Not all tools are available in all transformers.
FME Licensing Level
FME Professional edition and above
FME Community
The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.
Search for samples and information about this transformer on the FME Community.