microsoft / microsoft/SynapseML
Pyspark script does not terminate with SynapseML[BUG]
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
### SynapseML version
"com.microsoft.azure:synapseml_2.12:0.9.5-13-d1b51517-SNAPSHOT",
### System information
- **Language version**
- python 3.8, scala 2.12
- **Spark Version** (e.g. 3.2.3):
- **Spark Platform** (e.g. Dataproc):
### Describe the problem
The Spark script completes all the operation but the script is not terminated.
Even if I call spark.stop() the context is not stopped and the job does not terminate
### Code to reproduce issue
It's a normal fit operation with SynapseML, can't reporoduce the data anyway.
I have 100 M rows with ~30 features
### Other info / logs
_No response_
### What component(s) does this bug affect?
- [ ] `area/cognitive`: Cognitive project
- [ ] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [ ] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue
### What language(s) does this bug affect?
- [ ] `language/scala`: Scala source code
- [ ] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages
### What integration(s) does this bug affect?
- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations
Contributor guide
Assessment
This issue has not been assessed yet.