Getting 'Atum already initialized' error when trying to run Std+Conf multiple times
- Lingua principale
- Scala
- Stelle
- 33
- Fork
- 16
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Describe the bug
The error occurs when Standardization+Conformance jobs from a single Spark session. This happenned when running from Pramen since Pramen runs multiple Standardization+Conformance jobs in a single Spark Session.
## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Configure Pramen to run EnceladusSink for multiple tables in a pipeline.
2. Run the pipeline.
4. The error will occur saying "Control Measurements Tracking already initialized"
## Expected behavior
Allow running multiple Standardization+Conformance jobs multiple times from teh same Spask session.
## Screenshots
N/A
## Desktop (please complete the following information):
This was happening both in UAT and in PROD.
## Additional context
There is already logic in Enceladus that explicitly disables control metrics when conformance jobs runs after standardization. So single Standardization, Conformance, and Standardization+Conformance jobs run fine. The issue happens only if both Standardization and Conformance run multiple times.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.