KafkaConnector

Connects to an Apache Kafka cluster to send and receive messages.

Jump to Configuration

Typical Uses

  • Geoenrichment of data on a Kafka topic
  • Aggregation of data from Kafka topics and other message streams
  • Interoperability with Big Data analytics platforms

Usage Notes

  • The KafkaConnector connects to a list of bootstrap servers initially, but it may connect to further servers as they are discovered. It does not connect to the ZooKeeper instance directly.
  • The KafkaConnector allows for automatic or manual offset commits. For more information about this concept, see "Commits and Offsets" at https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html.

Configuration

Input Ports

Output Ports

Parameters

The remaining parameters available depend on the value of the Request > Action parameter. Parameters for each Action are detailed below.

Editing Transformer Parameters

Using a set of menu options, transformer parameters can be assigned by referencing other elements in the workspace. More advanced functions, such as an advanced editor and an arithmetic editor, are also available in some transformers. To access a menu of these options, click beside the applicable parameter. For more information, see Transformer Parameter Menu Options.

Defining Values

There are several ways to define a value for use in a Transformer. The simplest is to simply type in a value or string, which can include functions of various types such as attribute references, math and string functions, and workspace parameters. There are a number of tools and shortcuts that can assist in constructing values, generally available from the drop-down context menu adjacent to the value field.

Dialog Options - Tables

Transformers with table-style parameters have additional tools for populating and manipulating values.

Reference

Processing Behavior

Feature-Based

Feature Holding

No

Dependencies Apache Kafka
FME Licensing Level FME Base Edition and above
Aliases ApacheKafkaConnector
History Released FME 2019.0

FME Community

The FME Community is the place for demos, how-tos, articles, FAQs, and more. Get answers to your questions, learn from other users, and suggest, vote, and comment on new features.

Search for all results about the KafkaConnector on the FME Community.

 

Examples may contain information licensed under the Open Government Licence – Vancouver and/or the Open Government Licence – Canada.