XGBoost-Spark training fails sometimes due to "java.lang.NumberFormatException: For input string: "inf""

We are meeting this exception when running a regression training repeatly with the xgboost JVM jars built from the latest master branch using Scala 2.11, along with our spark example app jar.

This issue doesn't 100% occur, but nearly 50% rate under our test environment as below:
- **OS**: Ubuntu18.04,
- **CPU**: 12 cores
- **Memory**: 64G (Available ~40G)
- **Spark**: 2.4.3 with Hadhoop2.7, standalone mode with one worker on the same node.
- **Data**: 1.5G parquet files
- **Tree Method**: hist
- **numRound**: 100

The command is ` spark-submit --class ai.rapids.spark.examples.taxi.CPUMain --master spark://10.19.183.78:7077 --executor-memory 32G --jars /data/github/xgboost4j_2.11-1.0.0-SNAPSHOT.jar,/data/github/xgboost4j-spark_2.11-1.0.0-SNAPSHOT.jar /data/github/sample_xgboost_apps-0.1.4.jar -trainDataPath=/data/taxi/parquet/tr/ -evalDataPath=/data/taxi/parquet/test -format=parquet -showFeatures=0 -numWorkers=11 -treeMethod=hist -numRound=100` .

and error log follows

_19/09/11 17:29:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_0 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_3 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_1 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_5 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_4 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_6 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_10 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_9 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_2 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_7 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Putting block rdd_13_8 failed due to exception java.lang.NumberFormatException: For input string: "inf".
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_2 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_3 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_1 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_9 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_5 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_4 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_10 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_6 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_0 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_8 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 WARN BlockManager: Block rdd_13_7 could not be removed as it was not found on disk or in memory
19/09/11 17:31:50 ERROR Executor: Exception in task 9.0 in stage 3.0 (TID 24)
java.lang.NumberFormatException: For input string: "inf"
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at java.lang.Float.valueOf(Float.java:416)
at ml.dmlc.xgboost4j.java.Booster.evalSet(Booster.java:246)
at ml.dmlc.xgboost4j.java.XGBoost.train(XGBoost.java:202)
at ml.dmlc.xgboost4j.scala.XGBoost$.train(XGBoost.scala:64)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$.ml$dmlc$xgboost4j$scala$spark$XGBoost$$buildDistributedBooster(XGBoost.scala:330)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$$anonfun$ml$dmlc$xgboost4j$scala$spark$XGBoost$$trainForNonRanking$1.apply(XGBoost.scala:413)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$$anonfun$ml$dmlc$xgboost4j$scala$spark$XGBoost$$trainForNonRanking$1.apply(XGBoost.scala:409)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:337)
at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:335)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/09/11 17:31:50 ERROR Executor: Exception in task 7.0 in stage 3.0 (TID 22)
java.lang.NumberFormatException: For input string: "inf"
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at java.lang.Float.valueOf(Float.java:416)
at ml.dmlc.xgboost4j.java.Booster.evalSet(Booster.java:246)
at ml.dmlc.xgboost4j.java.XGBoost.train(XGBoost.java:202)
_
Concerning the jars and data files, pls refer to https://drive.google.com/open?id=1WKXyll4rmhwLU2qJTrLb6YYh-wsB8XQ3 .

For more on our example app, pls refer to https://github.com/rapidsai/spark-examples .