apache / apache/polaris

[FEATURE REQUEST] Support for OpenLineage

Open
#425 3 comments 10 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 17h
Merged PRs (30d)
137

Description

### Is your feature request related to a problem? Please describe.

[OpenLineage](https://openlineage.io/) is an open standard for reporting and collecting lineage information about processing jobs (i.e., which data sets were inputs to a processing job and which datasets were output). OpenLineage libraries are typically modeled as listeners or before/after hooks that are triggered by running processing jobs and have engine-specific code that collects information about the job and the datasets. That information is serialized as JSON and transmitted to a well-defined endpoint that either processes and stores that information or modifies and relays it to another endpoint.

### Describe the solution you'd like

Polaris is a good candidate for a proxying lineage endpoint because it has a canonical view of the datasets being processed and can augment the lineage payload with useful data. This is especially true when Polaris is used to access External catalogs, where the authoritative metadata lives somewhere else.

Spark or other OpenLineage clients can only report information about the datasets that can be gleaned from the client - e.g., the namespace of the data will be the Polaris endpoint that was used to access the data. The name of the catalog will be whatever name assigned to the catalog in that particular application (e.g., a user might configure the catalog as either `spark.sql.catalog.polaris` or `spark.sql.catalog.iceberg`). A table might have been renamed or moved from another catalog.

Polaris, however, knows exactly where the dataset originated and can use the table metadata's UUID field to uniquely identify the dataset. It also knows the snapshot information (datasets can be versioned in OpenLineage) as well as the schema, table properties, and other information that could be reported as an OpenLineage facet.

Polaris doesn't make sense as a container for lineage information, as parsing and storing that information is not cheap. However, there is already precedence for an [OpenLineage proxy](https://openlineage.io/docs/development/ol-proxy/), which can be used to augument the lineage information and pass it on to another service ([Marquez](https://github.com/MarquezProject/marquez) is the reference implementation of the OpenLineage server endpoint).

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing Polaris's catalog metadata and HTTP endpoint architecture alongside the OpenLineage proxy specification; done would require a scoped approach for receiving, augmenting, and forwarding lineage payloads, with its integration boundaries and tests identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.