databrickslabs / databrickslabs/ontos
[Feature]: Infer product lineage from platform lineage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 212
- Forks
- 71
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 43
Description
Is there an existing issue for this?
- I have searched the existing issues
Problem statement
Right now lineage is set manually for Data Products in the UI, by linking output to input ports.
Proposed Solution
We can infer relationsships by following the UC lineage information, where pipelines read tables and emit others. Give they are mapped to a contract already (could be via bulk import or manually) we can infer how they feed into each other, and be extension how products are linked.
Additional Context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are identified. Start by locating the existing Data Product UI lineage flow, contract mapping, and Unity Catalog lineage handling; define how pipeline table reads and outputs become product relationships and how completion will be verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100