microsoft / microsoft/SynapseML

LightGBMRegressor.fit gets timed out running via databricks-connect.

Open
#1,574 3 comments 0 reactions 1 assignee View on GitHub

@mhamilton723 is already working on this.

Since Aug 18, 2022.

area/lightgbm awaiting response
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

**Describe the bug**
`LightGBMRegressor.fit` gets stuck (times out) when running via databricks-connect.

**To Reproduce**
Fit an instance of `LightGBMRegressor` via databricks-connect.

**Expected behavior**
`fit` method is successfully completed.

**Info (please complete the following information):**
- SynapseML Version: 2.12
- SynapseML Coordinate: com.microsoft.azure:synapseml_2.12:0.9.5
- SynapseML Repository: https://mmlspark.azureedge.net/maven
- Spark Version: 3.2.1
- Scala Version: 2.12
- Databricks Version: 10.4 LTS
- Databricks-connect Version: 10.4.6

**Stacktrace**

```
2022-07-20T17:22:38.480+0000: [GC (Allocation Failure) [PSYoungGen: 524800K->21556K(611840K)] 524800K->21580K(2010112K), 0.0129807 secs] [Times: user=0.10 sys=0.00, real=0.01 secs]
2022-07-20T17:22:38.550+0000: [GC (Metadata GC Threshold) [PSYoungGen: 57016K->10494K(611840K)] 57040K->10526K(2010112K), 0.0049380 secs] [Times: user=0.08 sys=0.01, real=0.00 secs]
2022-07-20T17:22:38.555+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 10494K->0K(611840K)] [ParOldGen: 32K->9830K(636928K)] 10526K->9830K(1248768K), [Metaspace: 20515K->20515K(22528K)], 0.0180416 secs] [Times: user=0.19 sys=0.01, real=0.02 secs]
2022-07-20T17:22:39.567+0000: [GC (Metadata GC Threshold) [PSYoungGen: 263461K->12901K(611840K)] 273292K->22740K(1248768K), 0.0047544 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]
2022-07-20T17:22:39.572+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 12901K->0K(611840K)] [ParOldGen: 9838K->20825K(970240K)] 22740K->20825K(1582080K), [Metaspace: 34020K->34020K(36864K)], 0.0519531 secs] [Times: user=1.10 sys=0.04, real=0.05 secs]
2022-07-20T17:25:53.338+0000: [GC (Metadata GC Threshold) [PSYoungGen: 523080K->29927K(773120K)] 543906K->50761K(1743360K), 0.0166177 secs] [Times: user=0.13 sys=0.01, real=0.01 secs]
2022-07-20T17:25:53.355+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 29927K->0K(773120K)] [ParOldGen: 20833K->41112K(1268736K)] 50761K->41112K(2041856K), [Metaspace: 56284K->56213K(59392K)], 0.0775757 secs] [Times: user=1.39 sys=0.10, real=0.08 secs]
2022-07-20T17:26:22.914+0000: [GC (Allocation Failure) [PSYoungGen: 742912K->24067K(947200K)] 784024K->65195K(2215936K), 0.0142385 secs] [Times: user=0.23 sys=0.02, real=0.02 secs]
2022-07-20T17:26:23.235+0000: [GC (Metadata GC Threshold) [PSYoungGen: 475303K->33782K(1094144K)] 516431K->85437K(2362880K), 0.0151588 secs] [Times: user=0.34 sys=0.09, real=0.02 secs]
2022-07-20T17:26:23.250+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 33782K->0K(1094144K)] [ParOldGen: 51655K->68913K(1595904K)] 85437K->68913K(2690048K), [Metaspace: 81386K->72073K(100352K)], 0.0750877 secs] [Times: user=0.63 sys=0.20, real=0.07 secs]
2022-07-20T17:26:24.641+0000: [GC (Allocation Failure) [PSYoungGen: 1060352K->36634K(1105920K)] 1129265K->105556K(2701824K), 0.0215145 secs] [Times: user=0.34 sys=0.11, real=0.02 secs]
2022-07-20T17:26:28.141+0000: [GC (Allocation Failure) [PSYoungGen: 1096986K->49144K(1699840K)] 1165908K->156002K(3295744K), 0.0545540 secs] [Times: user=0.42 sys=0.22, real=0.05 secs]
2022-07-20T17:26:30.381+0000: [GC (Metadata GC Threshold) [PSYoungGen: 1145563K->73186K(1723904K)] 1252421K->184820K(3319808K), 0.0279235 secs] [Times: user=0.30 sys=0.22, real=0.03 secs]
2022-07-20T17:26:30.409+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 73186K->0K(1723904K)] [ParOldGen: 111634K->131221K(1966080K)] 184820K->131221K(3689984K), [Metaspace: 135096K->112661K(165888K)], 0.1516168 secs] [Times: user=1.30 sys=0.34, real=0.15 secs]
2022-07-20T17:26:31.571+0000: [GC (Allocation Failure) [PSYoungGen: 1650688K->60794K(2267648K)] 1781909K->192024K(4233728K), 0.0495676 secs] [Times: user=0.12 sys=0.13, real=0.05 secs]
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
```

**Additional context**
I am using DBR version 10.4 LTS. I want to use Synapse ML package with databricks-connect. I installed and attached Synapse ML to cluster and to be able to import with databricks-connect i also added Synapse to jar.packages. I am able to install and import it successfully. However when I try to run the fit method it times out at the workers side. Please see the error above. Would you have any idea how to solve this issue? Thanks

AB#1889707

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.