OpenLineage support
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
OpenLineage has become a common standard for collecting lineage metadata from processing engines. DataFusion is increasingly used to build query engines, but each DataFusion-based project currently needs to implement lineage extraction independently. This leads to duplicated effort and inconsistent OpenLineage support.
### Describe the solution you'd like
I would like DataFusion to expose OpenLineage support, either directly or through stable APIs/hooks that downstream engines can use.
Useful metadata to capture would include:
* Resolved input and output datasets
* Dataset schemas
* Column-level lineage, where possible
* Logical and/or physical plans, if appropriate
* Query metadata such as query ID, status, timing, and errors
I do not have a strong preference on the implementation. A separate crate, feature flag, or stable lineage extraction API would all be reasonable options.
### Describe alternatives you've considered
Each DataFusion-based engine could implement OpenLineage support independently by inspecting SQL, logical plans, or physical plans. However, this duplicates work, may depend on unstable internals, and can produce inconsistent lineage semantics.
### Additional context
OpenLineage integration would make DataFusion more useful as a foundation for production query engines and data platforms, especially for projects that want lineage and observability support without building it from scratch.
I am not very familiar with the DataFusion codebase yet, but I would be happy to collaborate with the DataFusion community on the OpenLineage side and help shape the expected metadata/modeling requirements.
Contributor guide
Research direction
Start by surveying DataFusion's SQL, logical-plan, and physical-plan APIs, then compare the available metadata with OpenLineage's dataset, schema, lineage, and query metadata requirements. Done means a decided integration scope—such as a separate crate, feature flag, or stable extraction API—with support boundaries documented and validated with the DataFusion community.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, data-engineering, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100