awslabs / awslabs/python-deequ
spark-submit job is not exiting until I hit ctrl+C if I use Check class
Open
enhancement
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
spark-submit job is not exiting until I hit ctrl+C if I use Check function
**Describe the solution you'd like**
Check function would start a self._spark_session.sparkContext._gateway server. If my code is completed, the server is still hang on that causes the spark job can't stop normally.
Adding a close function to class Check
def close(self):
self._spark_session.sparkContext._gateway.close()
return
Contributor guide
Assessment
This issue has not been assessed yet.