AbsaOSS / AbsaOSS/enceladus

Run has duplicate field - unique ID

Open
#1,102 0 comments 0 reactions 0 assignees View on GitHub
Menas priority: medium refactoring under discussion
Dominant language
Scala
Stars
33
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Background
For historical reasons the `Run`'s `uniqueId` field is duplicated in `controlMeasure.runUniqueId` as it represents the contents of the `_INFO` file. The `_INFO` file references the run's unique ID, so a `conformance` job can find the correct `standardization job` to continue from, as `standardization + conformance` together constitute a single run.

## Feature
We should reduce this noise and only keep one field.

## Proposed Solution [Optional]
Solution Ideas
1. We can remove the top level field `uniqueId` OR
1. We can not include the `controlMeasure.runUniqueId` from the `_INFO` file
1. ~~We might just use the `Spark application ID`~~
1. We should make this field non-optional
1. Maybe we can just have a separate file (e.g. `_MENAS_RUN`), written by `Standardization` and read by `Conformance` to link the two spark jobs that doesn't need to be propagated to publish.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.