dbt-labs / dbt-labs/dbt

[CT-3164] [Feature] add OpenLineage Datasets standard names in the Manifest

Open
#8,725 12 comments 12 reactions 0 assignees View on GitHub
engine:v1 type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

## Context
[OpenLineage](https://openlineage.io/) is an open framework for collecting lineage from data transformation tools.
OpenLineage primarily defines a [spec for exposing lineage in a standard format](https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md).
It also includes[ integrations for popular open source projects](https://github.com/OpenLineage/OpenLineage/tree/main/integration), including [Airflow](https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow), [Spark](https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark), [Flink](https://github.com/OpenLineage/OpenLineage/tree/main/integration/flink) and [dbt](https://github.com/OpenLineage/OpenLineage/tree/main/integration/dbt).
However, the best way to implement OpenLineage support is by making it built-in in the transformation tool itself. This way it is a lot more stable and reliable. For example, as of Airflow 2.7, [OpenLineage support is built-in operators](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-53+OpenLineage+in+Airflow)

## Proposal
We propose to add OpenLineage Datasets standard names in the dbt `Manifest`.
The [OpenLineage naming convention for Datasets](https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md) is what enables building a lineage graph across projects and technologies by creating a unique and canonical identifier for datasets.

Currently the [OpenLineage - dbt ](https://github.com/OpenLineage/OpenLineage/tree/main/integration/dbt) integration, must understand how dbt profiles work to be able to construct those unique identifiers.

Adding the OpenLineage standard namespaces and names in the Manifest will make the integration a lot more stable and not as prone to breaking when dbt implementation of profiles evolves.
This is a fairly small change that will make the OpenLineage support for dbt more reliable in a similar way that moving lineage to Airflow operators did in that context.

## Benefits
- reliable implementation of OpenLineage Dataset identifiers in dbt
- easier extensibility where each connector knows how to generate their dataset identifier
- easier support for custom connectors

### Describe alternatives you've considered

The current alternative is having the logic living in the openlineage-dbt integration.

The downsides of this approach are the following:
- OpenLineage needs [to understand the various ways profiles can be set in dbt ](https://github.com/OpenLineage/OpenLineage/blob/ba6f1d9b0c216deb7ecb227db297b7fd431ba323/integration/common/openlineage/common/provider/dbt/local.py#L194)which makes it dependent on internal implementation details.
- OpenLineage needs to know about the various connectors ([BigQuery](https://github.com/dbt-labs/dbt-bigquery), [Spark](https://github.com/dbt-labs/dbt-spark), ...)
- There's no good way to add support for custom dbt connectors

### Who will this benefit?

Users of dbt who want to benefit from the OpenLineage integration to send metadata to their favorite[ Data Catalog](https://openlineage.io/ecosystem) will benefit from a more reliable implementation and better coverage

### Are you interested in contributing this feature?

We can help contribute this feature but this requires input from dbt-core maintainers.

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.