AbsaOSS / AbsaOSS/enceladus

Getting 'Atum already initialized' error when trying to run Std+Conf multiple times

Aperta
#2,213 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
bug priority: undecided
Lingua principale
Scala
Stelle
33
Fork
16
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Describe the bug
The error occurs when Standardization+Conformance jobs from a single Spark session. This happenned when running from Pramen since Pramen runs multiple Standardization+Conformance jobs in a single Spark Session.

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Configure Pramen to run EnceladusSink for multiple tables in a pipeline.
2. Run the pipeline.
4. The error will occur saying "Control Measurements Tracking already initialized"

## Expected behavior
Allow running multiple Standardization+Conformance jobs multiple times from teh same Spask session.

## Screenshots
N/A

## Desktop (please complete the following information):
This was happening both in UAT and in PROD.

## Additional context
There is already logic in Enceladus that explicitly disables control metrics when conformance jobs runs after standardization. So single Standardization, Conformance, and Standardization+Conformance jobs run fine. The issue happens only if both Standardization and Conformance run multiple times.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.