AbsaOSS / AbsaOSS/enceladus

Run has duplicate field - unique ID

Offen
#1,102 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Menas priority: medium refactoring under discussion
Vorherrschende Sprache
Scala
Sterne
33
Forks
16
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Background
For historical reasons the `Run`'s `uniqueId` field is duplicated in `controlMeasure.runUniqueId` as it represents the contents of the `_INFO` file. The `_INFO` file references the run's unique ID, so a `conformance` job can find the correct `standardization job` to continue from, as `standardization + conformance` together constitute a single run.

## Feature
We should reduce this noise and only keep one field.

## Proposed Solution [Optional]
Solution Ideas
1. We can remove the top level field `uniqueId` OR
1. We can not include the `controlMeasure.runUniqueId` from the `_INFO` file
1. ~~We might just use the `Spark application ID`~~
1. We should make this field non-optional
1. Maybe we can just have a separate file (e.g. `_MENAS_RUN`), written by `Standardization` and read by `Conformance` to link the two spark jobs that doesn't need to be propagated to publish.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.