Getting 'Atum already initialized' error when trying to run Std+Conf multiple times
- Ngôn ngữ chính
- Scala
- Star
- 33
- Fork
- 16
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
## Describe the bug
The error occurs when Standardization+Conformance jobs from a single Spark session. This happenned when running from Pramen since Pramen runs multiple Standardization+Conformance jobs in a single Spark Session.
## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Configure Pramen to run EnceladusSink for multiple tables in a pipeline.
2. Run the pipeline.
4. The error will occur saying "Control Measurements Tracking already initialized"
## Expected behavior
Allow running multiple Standardization+Conformance jobs multiple times from teh same Spask session.
## Screenshots
N/A
## Desktop (please complete the following information):
This was happening both in UAT and in PROD.
## Additional context
There is already logic in Enceladus that explicitly disables control metrics when conformance jobs runs after standardization. So single Standardization, Conformance, and Standardization+Conformance jobs run fine. The issue happens only if both Standardization and Conformance run multiple times.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.