AbsaOSS / AbsaOSS/atum

Processing checkpoints and destructive operations on Dataframe

Open
#102 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
30
Forks
9
PR merge metrics
No merged PRs in 30d

Description

We need to come up with a way to process checkpoints between destructive operations. So if I do filtering of the data and lose some rows, I can flag the next checkpoint as "post-destructive", "deduplication" or something similar.

This will pay off a lot when we implement validation #95 and have the ability to attach the control measures onto data frame instead of spark session.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.