microsoft / microsoft/SynapseML

Spark pod not getting terminated after completion of spark job

Open
#1,471 4 comments 0 reactions 0 assignees View on GitHub
bug Spark Runtime
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

**Describe the bug**
Using SynpaseML package for applying LightGBM algorithm in a spark application. This spark application is running in the k8s environment. We have observed that even after the completion of all execution steps, the spark pod does not get terminated when the LightGBM package is used.

**Expected behavior**
After completion of all the steps, all the threads should be terminated and the spark pod should get terminated.

**Info (please complete the following information):**
- SynapseML Version: v0.9.4
- SynapseML Package: com.microsoft.azure:synapseml_2.12:0.9.4
- Spark Version 3.1.2
- Spark Platform Amazon k8s

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Reproduce the issue with SynapseML 0.9.4, Spark 3.1.2, LightGBM, and Amazon Kubernetes, then inspect which threads remain after the Spark job completes. Done means all threads terminate and the Spark pod exits normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, scala, spark
Domain
distributed-systems, infrastructure, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.