AbsaOSS / AbsaOSS/atum

Processing checkpoints and destructive operations on Dataframe

未關閉
#102 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Scala
星號
30
分支
9
PR 合併指標
30 天內沒有已合併 PR

描述

We need to come up with a way to process checkpoints between destructive operations. So if I do filtering of the data and lose some rows, I can flag the next checkpoint as "post-destructive", "deduplication" or something similar.

This will pay off a lot when we implement validation #95 and have the ability to attach the control measures onto data frame instead of spark session.

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。