FME can read attribute data stored in Delta Lake tables.
Delta Lake is an open-source storage layer that brings ACID (atomicity, consistency, isolation, durability) transaction capabilities to big data workloads. It is built on top of Apache Parquet and supports scalable, reliable data processing on data lakes. Delta Lake tables can be queried efficiently and incrementally, making them well-suited for analytical and batch processing use cases.
Delta Lake Product and System Requirements
|
Format |
FME Platform |
Operating System |
||||
|---|---|---|---|---|---|---|
|
Reader/Writer |
FME Form |
FME Flow |
FME Flow Hosted |
Windows 64-bit |
Linux |
Mac |
|
Reader |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Terminology: Delta Lake and FME
|
Delta Lake |
FME |
|
table |
feature type |
|
row |
feature |
|
column |
attribute |
Reader Overview
The Delta Lake reader reads a specified set of delta lake tables. Names and URLs of the tables must be defined in the connection parameters before they can be read.
Arbitrary WHERE clauses are fully supported per table.
Instead of using the Delta Lake reader directly, you can leverage its powerful SQL capabilities directly within the DatabaseQuerier transformer.
- For example, you can use the DatabaseQuerier to efficiently perform analysis over several different Delta Lake tables.
By incorporating Delta Lake within the DatabaseQuerier, you can enhance your data processing workflows, taking full advantage of SQL's flexibility and Delta Lake’s performance.