microsoft / microsoft/SynapseML
LightGBMRegressor.fit gets timed out running via databricks-connect.
@mhamilton723 is already working on this.
Since Aug 18, 2022.
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
**Describe the bug**
`LightGBMRegressor.fit` gets stuck (times out) when running via databricks-connect.
**To Reproduce**
Fit an instance of `LightGBMRegressor` via databricks-connect.
**Expected behavior**
`fit` method is successfully completed.
**Info (please complete the following information):**
- SynapseML Version: 2.12
- SynapseML Coordinate: com.microsoft.azure:synapseml_2.12:0.9.5
- SynapseML Repository: https://mmlspark.azureedge.net/maven
- Spark Version: 3.2.1
- Scala Version: 2.12
- Databricks Version: 10.4 LTS
- Databricks-connect Version: 10.4.6
**Stacktrace**
```
2022-07-20T17:22:38.480+0000: [GC (Allocation Failure) [PSYoungGen: 524800K->21556K(611840K)] 524800K->21580K(2010112K), 0.0129807 secs] [Times: user=0.10 sys=0.00, real=0.01 secs]
2022-07-20T17:22:38.550+0000: [GC (Metadata GC Threshold) [PSYoungGen: 57016K->10494K(611840K)] 57040K->10526K(2010112K), 0.0049380 secs] [Times: user=0.08 sys=0.01, real=0.00 secs]
2022-07-20T17:22:38.555+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 10494K->0K(611840K)] [ParOldGen: 32K->9830K(636928K)] 10526K->9830K(1248768K), [Metaspace: 20515K->20515K(22528K)], 0.0180416 secs] [Times: user=0.19 sys=0.01, real=0.02 secs]
2022-07-20T17:22:39.567+0000: [GC (Metadata GC Threshold) [PSYoungGen: 263461K->12901K(611840K)] 273292K->22740K(1248768K), 0.0047544 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]
2022-07-20T17:22:39.572+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 12901K->0K(611840K)] [ParOldGen: 9838K->20825K(970240K)] 22740K->20825K(1582080K), [Metaspace: 34020K->34020K(36864K)], 0.0519531 secs] [Times: user=1.10 sys=0.04, real=0.05 secs]
2022-07-20T17:25:53.338+0000: [GC (Metadata GC Threshold) [PSYoungGen: 523080K->29927K(773120K)] 543906K->50761K(1743360K), 0.0166177 secs] [Times: user=0.13 sys=0.01, real=0.01 secs]
2022-07-20T17:25:53.355+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 29927K->0K(773120K)] [ParOldGen: 20833K->41112K(1268736K)] 50761K->41112K(2041856K), [Metaspace: 56284K->56213K(59392K)], 0.0775757 secs] [Times: user=1.39 sys=0.10, real=0.08 secs]
2022-07-20T17:26:22.914+0000: [GC (Allocation Failure) [PSYoungGen: 742912K->24067K(947200K)] 784024K->65195K(2215936K), 0.0142385 secs] [Times: user=0.23 sys=0.02, real=0.02 secs]
2022-07-20T17:26:23.235+0000: [GC (Metadata GC Threshold) [PSYoungGen: 475303K->33782K(1094144K)] 516431K->85437K(2362880K), 0.0151588 secs] [Times: user=0.34 sys=0.09, real=0.02 secs]
2022-07-20T17:26:23.250+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 33782K->0K(1094144K)] [ParOldGen: 51655K->68913K(1595904K)] 85437K->68913K(2690048K), [Metaspace: 81386K->72073K(100352K)], 0.0750877 secs] [Times: user=0.63 sys=0.20, real=0.07 secs]
2022-07-20T17:26:24.641+0000: [GC (Allocation Failure) [PSYoungGen: 1060352K->36634K(1105920K)] 1129265K->105556K(2701824K), 0.0215145 secs] [Times: user=0.34 sys=0.11, real=0.02 secs]
2022-07-20T17:26:28.141+0000: [GC (Allocation Failure) [PSYoungGen: 1096986K->49144K(1699840K)] 1165908K->156002K(3295744K), 0.0545540 secs] [Times: user=0.42 sys=0.22, real=0.05 secs]
2022-07-20T17:26:30.381+0000: [GC (Metadata GC Threshold) [PSYoungGen: 1145563K->73186K(1723904K)] 1252421K->184820K(3319808K), 0.0279235 secs] [Times: user=0.30 sys=0.22, real=0.03 secs]
2022-07-20T17:26:30.409+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 73186K->0K(1723904K)] [ParOldGen: 111634K->131221K(1966080K)] 184820K->131221K(3689984K), [Metaspace: 135096K->112661K(165888K)], 0.1516168 secs] [Times: user=1.30 sys=0.34, real=0.15 secs]
2022-07-20T17:26:31.571+0000: [GC (Allocation Failure) [PSYoungGen: 1650688K->60794K(2267648K)] 1781909K->192024K(4233728K), 0.0495676 secs] [Times: user=0.12 sys=0.13, real=0.05 secs]
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
Received exception on call, retrying: java.net.ConnectException: Connection timed out (Connection timed out)
```
**Additional context**
I am using DBR version 10.4 LTS. I want to use Synapse ML package with databricks-connect. I installed and attached Synapse ML to cluster and to be able to import with databricks-connect i also added Synapse to jar.packages. I am able to install and import it successfully. However when I try to run the fit method it times out at the workers side. Please see the error above. Would you have any idea how to solve this issue? Thanks
AB#1889707
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.