Getting 'Atum already initialized' error when trying to run Std+Conf multiple times
- Dominant language
- Scala
- Stars
- 33
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
## Describe the bug
The error occurs when Standardization+Conformance jobs from a single Spark session. This happenned when running from Pramen since Pramen runs multiple Standardization+Conformance jobs in a single Spark Session.
## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Configure Pramen to run EnceladusSink for multiple tables in a pipeline.
2. Run the pipeline.
4. The error will occur saying "Control Measurements Tracking already initialized"
## Expected behavior
Allow running multiple Standardization+Conformance jobs multiple times from teh same Spask session.
## Screenshots
N/A
## Desktop (please complete the following information):
This was happening both in UAT and in PROD.
## Additional context
There is already logic in Enceladus that explicitly disables control metrics when conformance jobs runs after standardization. So single Standardization, Conformance, and Standardization+Conformance jobs run fine. The issue happens only if both Standardization and Conformance run multiple times.
Contributor guide
Assessment
This issue has not been assessed yet.