microsoft / microsoft/SynapseML
Spark pod not getting terminated after completion of spark job
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
**Describe the bug**
Using SynpaseML package for applying LightGBM algorithm in a spark application. This spark application is running in the k8s environment. We have observed that even after the completion of all execution steps, the spark pod does not get terminated when the LightGBM package is used.
**Expected behavior**
After completion of all the steps, all the threads should be terminated and the spark pod should get terminated.
**Info (please complete the following information):**
- SynapseML Version: v0.9.4
- SynapseML Package: com.microsoft.azure:synapseml_2.12:0.9.4
- Spark Version 3.1.2
- Spark Platform Amazon k8s
Contributor guide
Research direction
No source files or tests are named. Reproduce the issue with SynapseML 0.9.4, Spark 3.1.2, LightGBM, and Amazon Kubernetes, then inspect which threads remain after the Spark job completes. Done means all threads terminate and the Spark pod exits normally.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, scala, spark
- Domain
- distributed-systems, infrastructure, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100