[jvm-packages] XGBoost4J-Spark crashes with data having NaN value
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
when feature has NaN value like
label feature1
1 NaN
0 0.0
0 1.0
with the XGBoostClassifier setting of
`setMissing(Float.NaN)`
the training crashes with error message exit code 134. It looks like something related to the memory, but, in fact, it also fails to train even with three items of data I mentioned above.
However, when I use the setting of
`setTreeMethod("hist")`
problem solved.
here is the console output
```
19/07/25 17:46:49 ERROR XGBoostTaskFailedListener: Training Task Failed during XGBoost Training: ExecutorLostFailure(1,true,Some(Container marked as failed: *** on host: ***. Exit status: 134. Diagnostics: Exception from container-launch.
Container id: ***
Exit code: 134
Exception message: /bin/bash: line 1: 108718 Aborted LD_LIBRARY_PATH=*** -server -Xmx1024m -Djava.io.tmpdir=*** '-Dspark.authenticate=false' '-Dspark.network.crypto.enabled=false' '-Dspark.shuffle.service.port=***' '-Dspark.driver.port=***' -Dspark.yarn.app.container.log.dir=*** -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark:*** --executor-id 1 --hostname *** --cores 1 --app-id *** --user-class-path file:*** --user-class-path file:*** > ***> ***
Stack trace: ExitCodeException exitCode=134: /bin/bash: line 1: 108718 Aborted LD_LIBRARY_PATH=*** -server -Xmx1024m -Djava.io.tmpdir=*** '-Dspark.authenticate=false' '-Dspark.network.crypto.enabled=false' '-Dspark.shuffle.service.port=***' '-Dspark.driver.port=***' -Dspark.yarn.app.container.log.dir=*** -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark:*** --executor-id 1 --hostname *** --cores 1 --app-id *** --user-class-path file:*** --user-class-path file:*** > ***> ***
at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
at org.apache.hadoop.util.Shell.run(Shell.java:507)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 134
)), stopping SparkContext
19/07/25 17:46:49 WARN spark.ExecutorAllocationManager: Attempted to mark unknown executor 1 idle
19/07/25 17:46:49 ERROR java.RabitTracker: Uncaught exception thrown by worker:
org.apache.spark.SparkException: Job 6 cancelled because SparkContext was shut down
at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:837)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:835)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:835)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1848)
at org.apache.spark.util.EventLoop.stop(EventLoop.scala:83)
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1761)
at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1931)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1361)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1930)
at org.apache.spark.TaskFailedListener$$anon$1$$anonfun$run$1.apply$mcV$sp(SparkParallelismTracker.scala:131)
at org.apache.spark.TaskFailedListener$$anon$1$$anonfun$run$1.apply(SparkParallelismTracker.scala:131)
at org.apache.spark.TaskFailedListener$$anon$1$$anonfun$run$1.apply(SparkParallelismTracker.scala:131)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at org.apache.spark.TaskFailedListener$$anon$1.run(SparkParallelismTracker.scala:130)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:929)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:927)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:927)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$$anonfun$trainDistributed$1$$anon$1.run(XGBoost.scala:397)
19/07/25 17:46:52 ERROR java.RabitTracker: java.lang.InterruptedException: sleep interrupted
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.