awslabs / awslabs/python-deequ
AWS Glue Job runs indefinitely after exception occurs when adding Checks to the VerificationSuite
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- Avg merge
- 9d 22h
- Merged PRs (30d)
- 3
Description
**Describe the bug**
When running PyDeequ Checks on an AWS Glue job, if an exception occurs (e.g. accidentally calling `hasCompletenss(...)` instead of `hasCompleteness(...)`), the Glue job will keep running indefinitely.
**To Reproduce**
Glue 4.0
deequ-2.0.4-spark-3.3.jar
python-deequ 1.1.0
```
check_result = (
VerificationSuite(spark)
.onData(df)
.addCheck(
check.isComplete("primary_key")
.hasCompletenss("product_name", lambda x: x > 0.999) #notice the typo
)
.run()
```
**Expected behavior**
The expectation is that when errors such as this occur, the Glue job stops running or an exception is thrown so the callback server and spark session can be shut down:
**Screenshots**
In the CloudWatch Glue error logs you see that the job continues polling for executor status:
Contributor guide
Assessment
This issue has not been assessed yet.