microsoft / microsoft/SynapseML
LightGBMRegressor NetworkInit failed with exception on local port 12464
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
### mmlspark version 0.17
### step
1.add jars to `SPARK_HOME/jars`
```
-rw-r--r-- 1 wc wc 199947959 Jun 18 20:33 cntk-2.4.jar
-rw-r--r-- 1 wc wc 284184 Jun 18 20:33 commons-codec-1.10.jar
-rw-r--r-- 1 wc wc 61829 Jun 18 20:33 commons-logging-1.2.jar
-rw-r--r-- 1 wc wc 767140 Jun 18 20:33 httpclient-4.5.6.jar
-rw-r--r-- 1 wc wc 326356 Jun 18 20:33 httpcore-4.4.10.jar
-rw-r--r-- 1 wc wc 280515 Jun 18 20:33 jsch-0.1.54.jar
-rw-r--r-- 1 wc wc 2324062 Jun 18 20:33 lightgbmlib-2.2.350.jar
-rw-r--r-- 1 wc wc 4292898 Jun 18 20:33 mmlspark-0.17.jar
-rw-r--r-- 1 wc wc 65752623 Jun 18 20:33 opencv-3.2.0-1.jar
-rw-r--r-- 1 wc wc 5564007 Jun 18 20:33 scala-library-2.11.6.jar
-rw-r--r-- 1 wc wc 286389 Jun 18 20:33 spray-json_2.11-1.3.2.jar
```
2. run spark-shell
`spark-shell --master yarn`
3. my code
```
import org.apache.spark.ml.evaluation.RegressionEvaluator
import org.apache.spark.ml.feature.StringIndexer
import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder, TrainValidationSplit}
import org.apache.spark.ml.util.MLReadable
import org.apache.spark.ml.linalg.Vector
import org.apache.spark.sql.{Column, DataFrame}
import com.microsoft.ml.spark.LightGBMRegressor
import com.microsoft.ml.spark.LightGBMRegressionModel
import com.microsoft.ml.spark.LightGBMUtils
import com.microsoft.ml.spark.LightGBMConstants
val fileName = "hdfs://ttrms-nn-02.cars.com/user/wc/airfoil_self_noise.csv"
val labelColumnName = "scaled_sound_pressure_level"
val fileLocation = fileName
val dataset = spark.read.format("csv").option("header","true").option("delimiter",";").option("inferSchema","true").load(fileName).repartition(3)
val featuresColumn = "_features"
val rawPredCol = "rawPrediction"
dataset.show()
val lgbm = new LightGBMRegressor().setLabelCol(labelColumnName).setFeaturesCol(featuresColumn).setDefaultListenPort(LightGBMConstants.defaultLocalListenPort + 0) .setNumLeaves(5).setNumIterations(10)
val paramGrid = new ParamGridBuilder().addGrid(lgbm.numLeaves, Array(5, 10)).addGrid(lgbm.numIterations, Array(10, 20)).addGrid(lgbm.lambdaL1, Array(0.1, 0.5)).addGrid(lgbm.lambdaL2, Array(0.1, 0.5)).build()
val trainValidationSplit = new TrainValidationSplit().setEstimator(lgbm).setEvaluator(new RegressionEvaluator().setLabelCol(labelColumnName)).setEstimatorParamMaps(paramGrid).setTrainRatio(0.8).setParallelism(2)
val featurizer = LightGBMUtils.featurizeData(dataset, labelColumnName, featuresColumn)
val model = trainValidationSplit.fit(featurizer.transform(dataset))
model.transform(featurizer.transform(dataset))
assert(model != null)
// Validate lambda parameters set on model
val modelStr = model.bestModel.asInstanceOf[LightGBMRegressionModel].getModel.model
assert(modelStr.contains("[lambda_l1: 0.1]") || modelStr.contains("[lambda_l1: 0.5]"))
assert(modelStr.contains("[lambda_l2: 0.1]") || modelStr.contains("[lambda_l2: 0.5]"))
```
### driver output
```
19/06/22 14:37:28 INFO LightGBMRegressor: driver received socket from worker: 198.218.33.87:12464
19/06/22 14:37:28 INFO LightGBMRegressor: driver accepting a new connection...
19/06/22 15:07:16 INFO LightGBMRegressor: driver received socket from worker: 198.218.33.82:12432
19/06/22 15:07:16 INFO LightGBMRegressor: driver accepting a new connection...
```
### executor
```
19/06/22 15:07:00 INFO LightGBMRegressor: LightGBM worker got nodes for network init: null
19/06/22 15:07:00 INFO LightGBMRegressor: LightGBM worker listening on: 12464
19/06/22 15:07:00 INFO LightGBMRegressor: NetworkInit failed with exception on local port 12464, retrying: java.lang.NullPointerException
19/06/22 15:07:01 INFO LightGBMRegressor: NetworkInit failed with exception on local port 12464, retrying: java.lang.NullPointerException
19/06/22 15:07:03 INFO LightGBMRegressor: NetworkInit failed with exception on local port 12464, retrying: java.lang.NullPointerException
19/06/22 15:07:07 INFO LightGBMRegressor: NetworkInit failed with exception on local port 12464, retrying: java.lang.NullPointerException
19/06/22 15:07:15 ERROR Executor: Exception in task 1.0 in stage 7.0 (TID 10)
java.lang.NullPointerException
at com.microsoft.ml.spark.TrainUtils$.networkInit(TrainUtils.scala:275)
at com.microsoft.ml.spark.TrainUtils$.trainLightGBM(TrainUtils.scala:313)
at com.microsoft.ml.spark.LightGBMBase$$anonfun$4.apply(LightGBMBase.scala:49)
at com.microsoft.ml.spark.LightGBMBase$$anonfun$4.apply(LightGBMBase.scala:49)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
[driver.out.txt](https://github.com/Azure/mmlspark/files/3316649/driver.out.txt)
[excuter1.log](https://github.com/Azure/mmlspark/files/3316650/excuter1.log)
[excuter2.log](https://github.com/Azure/mmlspark/files/3316651/excuter2.log)
[airfoil_self_noise.csv.txt](https://github.com/Azure/mmlspark/files/3316653/airfoil_self_noise.csv.txt)
Contributor guide
Assessment
This issue has not been assessed yet.