• All Files

NLPTrainer

Trains a natural language processing (NLP) classification model based on the user’s specifications and the provided data.

NLPTrainer expects tagged data as input, with each feature bearing a single text and label. Some preprocessing of this learning data may be required, and the AttributeCreator transformer can be useful for this. Based on the set of learning data and the NLP features (specific types of information about the text) that the user specifies, a model will then be created and written to a *.fmd (FME MoDel) file. The companion transformer to this one, NLPClassifier, uses these *.fmd files to perform natural language classification, sorting texts into the categories labelled in the training data.

Configuration

Input Ports

ClosedInput

This transformer expects to receive a corpus of labelled texts in the form of features containing this information as attributes. All incoming features should have the same attribute names for each of the required pieces of information, both the label and the text. For example, if one of the input features uses 'my_label' as an attribute name for the label and 'my_text' as the attribute name for the text, all the other input features should use 'my_label' for labels and 'my_text' for texts as well.

Output Ports

ClosedSummary

This port will output one feature after the training process is over. The feature’s attributes will indicate the sizes of the test and training sets of data, as well as the accuracy of the model over the test set (on a scale of 0 to 1, where 1 is perfect accuracy), and some information about what NLP features the model finds helpful. (The exact structure of this information depends on the model type.) This information can be easily reviewed by connecting a logger.

Closed<Rejected>

Features which cannot be used to train the model, often because either the text itself or the label cannot be retrieved, are routed to this port.

Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.

Parameters

ClosedNLP Model
Model Type to Train This parameter specifies what kind of model the NLPTrainer will produce. All model types are suitable for use with the NLPClassifier transformer.
Label The attribute on incoming features which specifies the label.
Output Model Filename The save location for the model, in the form of an *.fmd file.
ClosedTraining and Testing
Text The attribute on incoming features which specifies the text.
Case Sensitive Determines whether the model will be case sensitive or not.
Data Percentage for Testing Determines what percentage of the input data will be used for testing the completed model. The remainder of the input data will be used for training.
Text Type Indicates whether this model is being trained to look at single-word or multiword texts.
NLP Features

Allows the user to tell the NLPTrainer what information about the text to take into account. Different NLP feature specifications will produce different models, which may vary considerably in their accuracy.

ClosedBeginning Characters
Number of Characters to Track

The number of characters from the beginning of each text to consider when creating the model. For example, if this value is 2, the first 2 characters of each text will be noted by the model.

Format attribute set: _nlp_chars_from_beginning
ClosedContains Common Words
Number of Words to Track

This parameter decides how many common words from the training corpus to keep track of. The words selected will always be the most common ones. For example, if the value is 500, the NLPTrainer will determine the 500 most common words throughout the training corpus and the model will pay attention to whether each of these words is used in each text.

Format attribute set: _nlp_number_words
Words to Ignore

This parameter, together with the next, allows the user to specify words that, however common, should not be tracked by the model. This is helpful, for example, for ignoring “stopwords”, words that are expected to be common throughout the texts regardless of category. This parameter offers a dropdown choice between no stopwords, FME’s list of English-language stopwords, or a custom list.

Format attribute set: _nlp_stopwords
Custom Stopword List Filename

If using a custom list of stopwords, or words to ignore, this parameter supplies the filepath for that list. The NLPTrainer expects this to be a newline-delimited plain text (*.txt) file.

Format attribute set: _nlp_stopwords_filename
ClosedContains Regular Expression
Expression to Match

This parameter provides a regular expression. When used, the model will track whether a text contains a match for this regular expression.

Format attribute set: _nlp_regex
ClosedContains String
String

When used, this NLP feature specification will cause the model to track whether each text contains the provided string.

Format attribute set: _nlp_contains_substring
ClosedEnding Characters
Number of Characters to Track

The number of characters from the end of each text to consider when creating the model. For example, if this value is 2, the last 2 characters of each text will be noted by the model.

Format attribute set: _nlp_chars_from_end
ClosedLength
Measure Length In

Determines whether the length of each text will be measured in characters, words, or sentences.

Format attribute set: _nlp_length_in
ClosedMean Length
Mean Length Of

Determines whether the model takes the mean length of sentences or words in the text.

Format attribute set: _nlp_mean_length_of
Mean Length In

Determines whether the model measures mean length of the specified components in words or characters.

Format attribute set: _nlp_mean_length_in

Additional References

For more information about natural language processing with FME, see the documentation for the companion NLPClassifier transformer.

Editing Transformer Parameters

Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.

ClosedHow to Set Parameter Values

Using the Text Editor

The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.

Text Editor

Using the Arithmetic Editor

The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.

Arithmetic Editor

Conditional Values

Set values depending on one or more test conditions that either pass or fail.

Parameter Condition Definition Dialog

Content

Expressions and strings can include a number of functions, characters, parameters, and more - whether entered directly in a parameter or constructed using one of the editors.

ClosedContent Types

String Functions

These functions manipulate and format strings.

Special Characters

A set of control characters is available in the Text Editor.

Math Functions

Math functions are available in both editors.

Math Operators

These operators are available in the Arithmetic Editor.

FME Feature Functions

These return primarily feature-specific values.

FME Parameters

FME and workspace-specific parameters may be used.
Working with User Parameters Create your own editable parameters.

Reference

ClosedRelated Transformers

NLPClassifier

Processing Behavior

Feature-Based

Feature Holding

Yes
Dependencies  
FME Licensing Level FME Base Edition and above
Aliases  
History Released FME 2019.0
Categories

Filters and Joins

Strings

FME Community

The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.

Search for all results about the NLPTrainer on the FME Community.

 

Examples may contain information licensed under the Open Government Licence – Vancouver