AbsaOSS / AbsaOSS/atum-service

Measurement Builder: perform calculations together / closer to each other, on a single DF pass ideally

未关闭
#98 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Agent enhancement
主要语言
Scala
星标
6
派生
1
平均合并
2 天 7 小时
30 天内合并 PR
4

描述

## Background
Currently, measurements are applied sequentially to a given dataframe. This might be not optimal - besides, this approach doesn't bring any benefit, it only slows things down. A better would be to somehow aggregate the measurements so that they get executed on a single pass through data. This might require implementing slightly different approach to `Measure.scala` using some kind of MeasurementBuilder class.

Except for Measure class, some other things might be changed here, see https://github.com/AbsaOSS/atum-service/pull/79#discussion_r1374159776 for example.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。