apache / apache/datafusion

OpenLineage support

Open
#22,944 1 comment 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

OpenLineage has become a common standard for collecting lineage metadata from processing engines. DataFusion is increasingly used to build query engines, but each DataFusion-based project currently needs to implement lineage extraction independently. This leads to duplicated effort and inconsistent OpenLineage support.

### Describe the solution you'd like

I would like DataFusion to expose OpenLineage support, either directly or through stable APIs/hooks that downstream engines can use.

Useful metadata to capture would include:
* Resolved input and output datasets
* Dataset schemas
* Column-level lineage, where possible
* Logical and/or physical plans, if appropriate
* Query metadata such as query ID, status, timing, and errors

I do not have a strong preference on the implementation. A separate crate, feature flag, or stable lineage extraction API would all be reasonable options.

### Describe alternatives you've considered

Each DataFusion-based engine could implement OpenLineage support independently by inspecting SQL, logical plans, or physical plans. However, this duplicates work, may depend on unstable internals, and can produce inconsistent lineage semantics.

### Additional context

OpenLineage integration would make DataFusion more useful as a foundation for production query engines and data platforms, especially for projects that want lineage and observability support without building it from scratch.

I am not very familiar with the DataFusion codebase yet, but I would be happy to collaborate with the DataFusion community on the OpenLineage side and help shape the expected metadata/modeling requirements.

Contributor guide

Open the contributing guide

Research direction

Start by surveying DataFusion's SQL, logical-plan, and physical-plan APIs, then compare the available metadata with OpenLineage's dataset, schema, lineage, and query metadata requirements. Done means a decided integration scope—such as a separate crate, feature flag, or stable extraction API—with support boundaries documented and validated with the DataFusion community.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, data-engineering, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.