AbsaOSS / AbsaOSS/enceladus

Run has duplicate field - unique ID

Abierto
#1,102 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Menas priority: medium refactoring under discussion
Lenguaje dominante
Scala
Estrellas
33
Forks
16
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Background
For historical reasons the `Run`'s `uniqueId` field is duplicated in `controlMeasure.runUniqueId` as it represents the contents of the `_INFO` file. The `_INFO` file references the run's unique ID, so a `conformance` job can find the correct `standardization job` to continue from, as `standardization + conformance` together constitute a single run.

## Feature
We should reduce this noise and only keep one field.

## Proposed Solution [Optional]
Solution Ideas
1. We can remove the top level field `uniqueId` OR
1. We can not include the `controlMeasure.runUniqueId` from the `_INFO` file
1. ~~We might just use the `Spark application ID`~~
1. We should make this field non-optional
1. Maybe we can just have a separate file (e.g. `_MENAS_RUN`), written by `Standardization` and read by `Conformance` to link the two spark jobs that doesn't need to be propagated to publish.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.