NLPTrainer

Categories

Filters and Joins

Strings

Related Transformers

AzureTextAnalyticsConnector

NLPClassifier

Trains a natural language processing (NLP) classification model based on the user’s specifications and the provided data.

NLPTrainer expects tagged data as input, with each feature bearing a single text and label. Some preprocessing of this learning data may be required, and the AttributeCreator transformer can be useful for this. Based on the set of learning data and the NLP features (specific types of information about the text) that the user specifies, a model will then be created and written to a *.fmd (FME MoDel) file. The companion transformer to this one, NLPClassifier, uses these *.fmd files to perform natural language classification, sorting texts into the categories labelled in the training data.

Usage Notes

  • For more information about natural language processing with FME, see the documentation for the companion NLPClassifier transformer.

Configuration

Input Ports

This transformer expects to receive a corpus of labelled texts in the form of features containing this information as attributes. All incoming features should have the same attribute names for each of the required pieces of information, both the label and the text. For example, if one of the input features uses 'my_label' as an attribute name for the label and 'my_text' as the attribute name for the text, all the other input features should use 'my_label' for labels and 'my_text' for texts as well.

Output Ports

This port will output one feature after the training process is over. The feature’s attributes will indicate the sizes of the test and training sets of data, as well as the accuracy of the model over the test set (on a scale of 0 to 1, where 1 is perfect accuracy), and some information about what NLP features the model finds helpful. (The exact structure of this information depends on the model type.) This information can be easily reviewed by connecting a logger.

Features which cannot be used to train the model, often because either the text itself or the label cannot be retrieved, are routed to this port.

Rejected Feature Handling: can be set to either terminate the translation or continue running when it encounters a rejected feature. This setting is available both as a default FME option and as a workspace parameter.

Parameters

Model Type to Train

This parameter specifies what kind of model the NLPTrainer will produce. All model types are suitable for use with the NLPClassifier transformer.

Label

The attribute on incoming features which specifies the label.

Output Model Filename

The save location for the model, in the form of an *.fmd file.

Text

The attribute on incoming features which specifies the text.

Case Sensitive

Determines whether the model will be case sensitive or not.

Data Percentage for Testing

Determines what percentage of the input data will be used for testing the completed model. The remainder of the input data will be used for training.

Text Type

Indicates whether this model is being trained to look at single-word or multiword texts.

NLP Features

Allows the user to tell the NLPTrainer what information about the text to take into account. Different NLP feature specifications will produce different models, which may vary considerably in their accuracy.

Number of Characters to Track

The number of characters from the beginning of each text to consider when creating the model. For example, if this value is 2, the first 2 characters of each text will be noted by the model.

Format attribute set: _nlp_chars_from_beginning

Number of Words to Track

This parameter decides how many common words from the training corpus to keep track of. The words selected will always be the most common ones. For example, if the value is 500, the NLPTrainer will determine the 500 most common words throughout the training corpus and the model will pay attention to whether each of these words is used in each text.

Format attribute set: _nlp_number_words

Words to Ignore

This parameter, together with the next, allows the user to specify words that, however common, should not be tracked by the model. This is helpful, for example, for ignoring “stopwords”, words that are expected to be common throughout the texts regardless of category. This parameter offers a dropdown choice between no stopwords, FME’s list of English-language stopwords, or a custom list.

Format attribute set: _nlp_stopwords

Custom Stopword List Filename

If using a custom list of stopwords, or words to ignore, this parameter supplies the filepath for that list. The NLPTrainer expects this to be a newline-delimited plain text (*.txt) file.

Format attribute set: _nlp_stopwords_filename

Expression to Match

This parameter provides a regular expression. When used, the model will track whether a text contains a match for this regular expression.

Format attribute set: _nlp_regex

String

When used, this NLP feature specification will cause the model to track whether each text contains the provided string.

Format attribute set: _nlp_contains_substring

Number of Characters to Track

The number of characters from the end of each text to consider when creating the model. For example, if this value is 2, the last 2 characters of each text will be noted by the model.

Format attribute set: _nlp_chars_from_end

Measure Length In

Determines whether the length of each text will be measured in characters, words, or sentences.

Format attribute set: _nlp_length_in

Mean Length Of

Determines whether the model takes the mean length of sentences or words in the text.

Format attribute set: _nlp_mean_length_of

Mean Length In

Determines whether the model measures mean length of the specified components in words or characters.

Format attribute set: _nlp_mean_length_in

Editing Transformer Parameters

Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.

Using the Text Editor

The Text Editor provides a convenient way to construct text strings (including regular expressions) from various data sources, such as attributes, parameters, and constants, where the result is used directly inside a parameter.

Text Editor

Using the Arithmetic Editor

The Arithmetic Editor provides a convenient way to construct math expressions from various data sources, such as attributes, parameters, and feature functions, where the result is used directly inside a parameter.

Arithmetic Editor

Conditional Values

Set values depending on one or more test conditions that either pass or fail.

Parameter Condition Definition Dialog

Content

Expressions and strings can include a number of functions, characters, parameters, and more.

When setting values - whether entered directly in a parameter or constructed using one of the editors - strings and expressions containing String, Math, Date/Time or FME Feature Functions will have those functions evaluated. Therefore, the names of these functions (in the form @<function_name>) should not be used as literal string values.

String Functions

These functions manipulate and format strings.

Special Characters

 A set of control characters is available in the Text Editor.

Math Functions

Math functions are available in both editors.
Date/Time Functions Date and time functions are available in the Text Editor.

Math Operators

These operators are available in the Arithmetic Editor.

FME Feature Functions

These return primarily feature-specific values.

FME Parameters

FME and workspace-specific parameters may be used.
Creating and Modifying User Parameters Create your own editable parameters.

Dialog Options - Tables

Transformers with table-style parameters have additional tools for populating and manipulating values.

Row Reordering

Enabled once you have clicked on a row item. Choices include:

  • Add a row
  • Remove a row
  • Move current row up one
  • Move current row down one
  • Move current row to top
  • Move current row to bottom

Cut, Copy, and Paste

Enabled once you have clicked on a row item. Choices include:

  • Cut a row - delete and copy to clipboard
  • Copy a row to the clipboard
  • Paste a row from the clipboard

Cut, copy, and paste may be used within a transformer, or between transformers.

Filter

Start typing a string, and the matrix will only display rows matching those characters. Searches all columns. This only affects the display of attributes within the transformer - it does not alter which attributes are output.

Import

Import populates the table with a set of new attributes read from a dataset. Specific application varies between transformers.

Reset/Refresh

Generally resets the table to its initial state, and may provide additional options to remove invalid entries. Behavior varies between transformers.

Note: Not all tools are available in all transformers.

Reference

Processing Behavior

Feature-Based

Feature Holding

Yes
Dependencies  
Aliases  
History Released FME 2019.0

FME Community

The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.

Search for all results about the NLPTrainer on the FME Community.

 

Examples may contain information licensed under the Open Government Licence – Vancouver and/or the Open Government Licence – Canada.