microsoft / microsoft/SynapseML

Pyspark script does not terminate with SynapseML[BUG]

Open
#1,762 3 comments 0 reactions 1 assignee Claimed by @mhamilton723 View on GitHub
awaiting response bug
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

### SynapseML version

"com.microsoft.azure:synapseml_2.12:0.9.5-13-d1b51517-SNAPSHOT",

### System information

- **Language version**
- python 3.8, scala 2.12
- **Spark Version** (e.g. 3.2.3):
- **Spark Platform** (e.g. Dataproc):

### Describe the problem

The Spark script completes all the operation but the script is not terminated.
Even if I call spark.stop() the context is not stopped and the job does not terminate

### Code to reproduce issue

It's a normal fit operation with SynapseML, can't reporoduce the data anyway.

I have 100 M rows with ~30 features

### Other info / logs

_No response_

### What component(s) does this bug affect?

- [ ] `area/cognitive`: Cognitive project
- [ ] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [ ] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue

### What language(s) does this bug affect?

- [ ] `language/scala`: Scala source code
- [ ] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.