[CT-3164] [Feature] add OpenLineage Datasets standard names in the Manifest
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this your first time submitting a feature request?
- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
## Context
[OpenLineage](https://openlineage.io/) is an open framework for collecting lineage from data transformation tools.
OpenLineage primarily defines a [spec for exposing lineage in a standard format](https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md).
It also includes[ integrations for popular open source projects](https://github.com/OpenLineage/OpenLineage/tree/main/integration), including [Airflow](https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow), [Spark](https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark), [Flink](https://github.com/OpenLineage/OpenLineage/tree/main/integration/flink) and [dbt](https://github.com/OpenLineage/OpenLineage/tree/main/integration/dbt).
However, the best way to implement OpenLineage support is by making it built-in in the transformation tool itself. This way it is a lot more stable and reliable. For example, as of Airflow 2.7, [OpenLineage support is built-in operators](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-53+OpenLineage+in+Airflow)
## Proposal
We propose to add OpenLineage Datasets standard names in the dbt `Manifest`.
The [OpenLineage naming convention for Datasets](https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md) is what enables building a lineage graph across projects and technologies by creating a unique and canonical identifier for datasets.
Currently the [OpenLineage - dbt ](https://github.com/OpenLineage/OpenLineage/tree/main/integration/dbt) integration, must understand how dbt profiles work to be able to construct those unique identifiers.
Adding the OpenLineage standard namespaces and names in the Manifest will make the integration a lot more stable and not as prone to breaking when dbt implementation of profiles evolves.
This is a fairly small change that will make the OpenLineage support for dbt more reliable in a similar way that moving lineage to Airflow operators did in that context.
## Benefits
- reliable implementation of OpenLineage Dataset identifiers in dbt
- easier extensibility where each connector knows how to generate their dataset identifier
- easier support for custom connectors
### Describe alternatives you've considered
The current alternative is having the logic living in the openlineage-dbt integration.
The downsides of this approach are the following:
- OpenLineage needs [to understand the various ways profiles can be set in dbt ](https://github.com/OpenLineage/OpenLineage/blob/ba6f1d9b0c216deb7ecb227db297b7fd431ba323/integration/common/openlineage/common/provider/dbt/local.py#L194)which makes it dependent on internal implementation details.
- OpenLineage needs to know about the various connectors ([BigQuery](https://github.com/dbt-labs/dbt-bigquery), [Spark](https://github.com/dbt-labs/dbt-spark), ...)
- There's no good way to add support for custom dbt connectors
### Who will this benefit?
Users of dbt who want to benefit from the OpenLineage integration to send metadata to their favorite[ Data Catalog](https://openlineage.io/ecosystem) will benefit from a more reliable implementation and better coverage
### Are you interested in contributing this feature?
We can help contribute this feature but this requires input from dbt-core maintainers.
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.