AbsaOSS / AbsaOSS/atum

Atum redesign

Đang mở
#28 6 bình luận 2 reaction 0 người được giao Xem trên GitHub
under discussion
Ngôn ngữ chính
Scala
Star
30
Fork
9
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

## Background
Currently, Atum relies on the global state of a Spark Application. This complicates the usage of Atum for jobs that are slightly more complicated than just a pipeline of a single dataframe. If there are several dataframes and several reads/writes and not every read and write is associated with control measurements, Atum will try to process all dataframes as if all require measurements.

The current workaround for such use cases is `disableControlMeasuresTracking()` method that is invoked before writing a dataframe that does not require control measurements.

## Feature
- [ ] Control measurements should be attached to a dataframe, not to the Spark session. E.g., to turn on control measurements users should do `df.enableControlMeasuresTracking()` instead of `spark. enableControlMeasuresTracking()`. Same for switching off control measurements.
- [ ] The measurements should happen to the dataframe it was initialized and the derived ones. Other dataframes shouldn't be affected.
- [ ] Checkpoints and other housekeeping information should not be kept in the global state.
- [ ] Adding metadata should be done as dataframe implicits (e.g. `df.setAdditionalInfo(...)`).
- [ ] Atum should keep checkpoints for each registered dataframe separately.
- [ ] Atum plugins should have an event that guaranteed to be sent last. Atum should guarantee that no more events are sent after that.

## Additonal context
After the new design is confirmed this issue can be converted to epic and all subitems to tasks.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.