Automatic Column lineage Tracking for Hudi Tables
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**What the feature achieves:**
The feature is to integrate Hudi with OpenLineage for automatic dataset-level and column-level lineage tracking.
**Why this feature is needed:**
Hudi uses custom Relation objects like IncrementalRelation, MergeOnReadIncrementalRelation and also does transformation from Dataframe to JavaRDD in StreamSync due to that lineage tracking information cannot be parsed through the Spark's Logical or Physical plans. Because of this gap, lineage systems cannot reliably capture how Hudi target columns are derived from source datasets and columns.
So, I am proposing to integrate with Open lineage.
This feature would add the missing Hudi integration points needed so OpenLineage-compatible systems can collect lineage automatically.
### User Experience
**How users will use this feature:**
Users run existing Hudi SQL writes or HoodieStreamer jobs, and lineage metadata is emitted automatically to the configured lineage system.
#### Usage examples
**Example 1: Spark SQL**
INSERT INTO target_table
SELECT id, amount * 100 AS amount_cents
FROM source_table;
Expected lineage:
- target_table.id <- source_table.id
- target_table.amount_cents <- source_table.amount
**Example 2: HoodieStreamer**
A HoodieStreamer pipeline with a custom Transformer should emit lineage between source_table columns and any dataset on which joins are applied as part of Transformer and the target_table columns.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.