awslabs / awslabs/python-deequ

AWS Glue Job runs indefinitely after exception occurs when adding Checks to the VerificationSuite

Open
#156 2 comments 0 reactions 0 assignees View on GitHub
bug researching
Dominant language
Jupyter Notebook
Stars
826
Forks
158
Avg merge
9d 22h
Merged PRs (30d)
3

Description

**Describe the bug**
When running PyDeequ Checks on an AWS Glue job, if an exception occurs (e.g. accidentally calling `hasCompletenss(...)` instead of `hasCompleteness(...)`), the Glue job will keep running indefinitely.

**To Reproduce**
Glue 4.0
deequ-2.0.4-spark-3.3.jar
python-deequ 1.1.0
```
check_result = (
VerificationSuite(spark)
.onData(df)
.addCheck(
check.isComplete("primary_key")
.hasCompletenss("product_name", lambda x: x > 0.999) #notice the typo
)
.run()
```
**Expected behavior**
The expectation is that when errors such as this occur, the Glue job stops running or an exception is thrown so the callback server and spark session can be shut down:

**Screenshots**
In the CloudWatch Glue error logs you see that the job continues polling for executor status:
image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.