AbsaOSS / AbsaOSS/atum-service

Measurement Builder: perform calculations together / closer to each other, on a single DF pass ideally

Offen
#98 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Agent enhancement
Vorherrschende Sprache
Scala
Sterne
6
Forks
1
Ø Merge
2 T. 7 Std.
Gemergte PRs (30 T.)
4

Beschreibung

## Background
Currently, measurements are applied sequentially to a given dataframe. This might be not optimal - besides, this approach doesn't bring any benefit, it only slows things down. A better would be to somehow aggregate the measurements so that they get executed on a single pass through data. This might require implementing slightly different approach to `Measure.scala` using some kind of MeasurementBuilder class.

Except for Measure class, some other things might be changed here, see https://github.com/AbsaOSS/atum-service/pull/79#discussion_r1374159776 for example.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.