awslabs / awslabs/python-deequ

spark-submit job is not exiting until I hit ctrl+C if I use Check class

Open
#7 7 comments 2 reactions 1 assignee Claimed by @jaoanan1126 View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
826
Forks
158
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

spark-submit job is not exiting until I hit ctrl+C if I use Check function

**Describe the solution you'd like**

Check function would start a self._spark_session.sparkContext._gateway server. If my code is completed, the server is still hang on that causes the spark job can't stop normally.

Adding a close function to class Check

def close(self):
self._spark_session.sparkContext._gateway.close()
return

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.