dmlc / dmlc/xgboost

[jvm-packages] XGBoost4J-Spark crashes with data having NaN value

Open
#4,714 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

when feature has NaN value like
label feature1
1 NaN
0 0.0
0 1.0

with the XGBoostClassifier setting of

`setMissing(Float.NaN)`

the training crashes with error message exit code 134. It looks like something related to the memory, but, in fact, it also fails to train even with three items of data I mentioned above.

However, when I use the setting of

`setTreeMethod("hist")`

problem solved.

here is the console output

```
19/07/25 17:46:49 ERROR XGBoostTaskFailedListener: Training Task Failed during XGBoost Training: ExecutorLostFailure(1,true,Some(Container marked as failed: *** on host: ***. Exit status: 134. Diagnostics: Exception from container-launch.
Container id: ***
Exit code: 134
Exception message: /bin/bash: line 1: 108718 Aborted LD_LIBRARY_PATH=*** -server -Xmx1024m -Djava.io.tmpdir=*** '-Dspark.authenticate=false' '-Dspark.network.crypto.enabled=false' '-Dspark.shuffle.service.port=***' '-Dspark.driver.port=***' -Dspark.yarn.app.container.log.dir=*** -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark:*** --executor-id 1 --hostname *** --cores 1 --app-id *** --user-class-path file:*** --user-class-path file:*** > ***> ***

Stack trace: ExitCodeException exitCode=134: /bin/bash: line 1: 108718 Aborted LD_LIBRARY_PATH=*** -server -Xmx1024m -Djava.io.tmpdir=*** '-Dspark.authenticate=false' '-Dspark.network.crypto.enabled=false' '-Dspark.shuffle.service.port=***' '-Dspark.driver.port=***' -Dspark.yarn.app.container.log.dir=*** -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark:*** --executor-id 1 --hostname *** --cores 1 --app-id *** --user-class-path file:*** --user-class-path file:*** > ***> ***
at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
at org.apache.hadoop.util.Shell.run(Shell.java:507)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Container exited with a non-zero exit code 134
)), stopping SparkContext
19/07/25 17:46:49 WARN spark.ExecutorAllocationManager: Attempted to mark unknown executor 1 idle
19/07/25 17:46:49 ERROR java.RabitTracker: Uncaught exception thrown by worker:
org.apache.spark.SparkException: Job 6 cancelled because SparkContext was shut down
at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:837)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:835)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:835)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1848)
at org.apache.spark.util.EventLoop.stop(EventLoop.scala:83)
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1761)
at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1931)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1361)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1930)
at org.apache.spark.TaskFailedListener$$anon$1$$anonfun$run$1.apply$mcV$sp(SparkParallelismTracker.scala:131)
at org.apache.spark.TaskFailedListener$$anon$1$$anonfun$run$1.apply(SparkParallelismTracker.scala:131)
at org.apache.spark.TaskFailedListener$$anon$1$$anonfun$run$1.apply(SparkParallelismTracker.scala:131)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at org.apache.spark.TaskFailedListener$$anon$1.run(SparkParallelismTracker.scala:130)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:929)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:927)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:927)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$$anonfun$trainDistributed$1$$anon$1.run(XGBoost.scala:397)
19/07/25 17:46:52 ERROR java.RabitTracker: java.lang.InterruptedException: sleep interrupted
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.