apache / apache/hudi

Automatic Column lineage Tracking for Hudi Tables

Open
#18,298 7 comments 0 reactions 1 assignee Claimed by @vinothchandar View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**What the feature achieves:**
The feature is to integrate Hudi with OpenLineage for automatic dataset-level and column-level lineage tracking.

**Why this feature is needed:**
Hudi uses custom Relation objects like IncrementalRelation, MergeOnReadIncrementalRelation and also does transformation from Dataframe to JavaRDD in StreamSync due to that lineage tracking information cannot be parsed through the Spark's Logical or Physical plans. Because of this gap, lineage systems cannot reliably capture how Hudi target columns are derived from source datasets and columns.

So, I am proposing to integrate with Open lineage.
This feature would add the missing Hudi integration points needed so OpenLineage-compatible systems can collect lineage automatically.

### User Experience

**How users will use this feature:**
Users run existing Hudi SQL writes or HoodieStreamer jobs, and lineage metadata is emitted automatically to the configured lineage system.

#### Usage examples
**Example 1: Spark SQL**
INSERT INTO target_table
SELECT id, amount * 100 AS amount_cents
FROM source_table;
Expected lineage:
- target_table.id <- source_table.id
- target_table.amount_cents <- source_table.amount

**Example 2: HoodieStreamer**
A HoodieStreamer pipeline with a custom Transformer should emit lineage between source_table columns and any dataset on which joins are applied as part of Transformer and the target_table columns.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.