AbsaOSS / AbsaOSS/spline-spark-agent

Capturing Lineage when Data is not Written

Đang mở
#132 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Scala
Star
210
Fork
102
Merge trung bình
1 ngày 1 giờ
Pull request đã merge (30 ngày)
1

Mô tả

I have a question on capturing lineage for operations that are not Written, but reads are still performed.

Take for example this code:
`result = spark.sql("select * from hive.table limit 1").collect()`

Has a materialized dataframe, which could be used to parameterize subsequent queries, or produce reports or datasets that may be written to resources outside spark, but I don't believe lineage is captured via Spline.

We have quite a few use cases where we want to capture lineage in this scenario and want to gauge what it may take for these lineage records to flow to the `LineageDispatcher`. We are using a custom LineageDispatcher for our case as well. Additionally we will be adding additional "Meta Information" to these lineage records so that they still have "Attribution" to the Job(code) or User who performed the read or write.

Thanks, I understand why Spline doesn't capture Lineage without a write, as it's a Dataset to Dataset lineage tool, but want to start a discussion to understand what may be needed to enable this, or if it's realistic in general.

Thanks for the advice.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.