Failed to connect to the tracker

Open

#11,511 opened on Jun 13, 2025 3 comments 0 reactions 0 assignees View on GitHub

status: need update

Dominant language: C++
Stars: 28.8k
Forks: 8.9k
Avg merge: 1d 12h
Merged PRs (30d): 54

Description

When running an XGBoost4J job on Spark with YARN, I encountered an error related to the communication group initialization. The job fails with XGBoostError: Failed to bootstrap the communication group.

Environment:

XGBoost version: 3..0.2
Spark version: 3.5.6
Hadoop version:3.3.6
Cluster manager: YARN
Operating system:Ubuntu 24
Java version:11
XGBoost training mode: distributed

logs:

```
25/06/13 19:31:12 INFO YarnScheduler: Successfully scheduled all the 4 tasks for barrier stage 10.
25/06/13 19:31:12 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on node131:36743 (size: 47.9 KiB, free: 4.6 GiB)
25/06/13 19:31:12 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on node133:44851 (size: 47.9 KiB, free: 4.6 GiB)
25/06/13 19:31:12 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on node131:33239 (size: 47.9 KiB, free: 4.6 GiB)
25/06/13 19:31:13 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on node133:44851 (size: 36.9 KiB, free: 4.6 GiB)
25/06/13 19:31:13 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on node131:36743 (size: 36.9 KiB, free: 4.6 GiB)
25/06/13 19:31:13 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on node131:33239 (size: 36.9 KiB, free: 4.6 GiB)
25/06/13 19:31:13 WARN TaskSetManager: Lost task 3.0 in stage 10.0 (TID 19) (node133 executor 3): ml.dmlc.xgboost4j.java.XGBoostError: [11:31:24] /workspace/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:186: [11:31:24] /workspace/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:89: Check failed: jenv->ExceptionOccurred():
Stack trace:
[bt] (0) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7de4f9ddf58e]
[bt] (1) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(XGBoost4jCallbackDataIterNext+0xb5b) [0x7de4f9dd862b]
[bt] (2) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::IteratorAdapter::Next()+0x1b) [0x7de4fa0c559b]
[bt] (3) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::SimpleDMatrix::SimpleDMatrix >(xgboost::data::IteratorAdapter*, float, int, xgboost::DataSplitMode)+0x2c3) [0x7de4fa23ce73]
[bt] (4) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::DMatrix* xgboost::DMatrix::Create >(xgboost::data::IteratorAdapter*, float, int, std::string const&, xgboost::DataSplitMode)+0x3e) [0x7de4fa1a732e]
[bt] (5) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(XGDMatrixCreateFromDataIter+0x145) [0x7de4f9df01b5]
[bt] (6) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromDataIter+0x229) [0x7de4f9dd4829]
[bt] (7) [0x7de52580f710]

Stack trace:
[bt] (0) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7de4f9ddf58e]
[bt] (1) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(+0x19f20b) [0x7de4f9d9f20b]
[bt] (2) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::IteratorAdapter::Next()+0x1b) [0x7de4fa0c559b]
[bt] (3) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::SimpleDMatrix::SimpleDMatrix >(xgboost::data::IteratorAdapter*, float, int, xgboost::DataSplitMode)+0x2c3) [0x7de4fa23ce73]
[bt] (4) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::DMatrix* xgboost::DMatrix::Create >(xgboost::data::IteratorAdapter*, float, int, std::string const&, xgboost::DataSplitMode)+0x3e) [0x7de4fa1a732e]
[bt] (5) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(XGDMatrixCreateFromDataIter+0x145) [0x7de4f9df01b5]
[bt] (6) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromDataIter+0x229) [0x7de4f9dd4829]
[bt] (7) [0x7de52580f710]

at ml.dmlc.xgboost4j.java.XGBoostJNI.checkCall(XGBoostJNI.java:49)
at ml.dmlc.xgboost4j.java.DMatrix.(DMatrix.java:60)
at ml.dmlc.xgboost4j.scala.DMatrix.(DMatrix.scala:46)
at ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator.ml$dmlc$xgboost4j$scala$spark$XGBoostEstimator$$buildDMatrix$1(XGBoostEstimator.scala:326)
at ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator$$anon$2.next(XGBoostEstimator.scala:354)
at ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator$$anon$2.next(XGBoostEstimator.scala:350)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$.$anonfun$train$4(XGBoost.scala:259)
at ml.dmlc.xgboost4j.scala.spark.Utils$.withResource(Utils.scala:119)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$.$anonfun$train$2(XGBoost.scala:258)
at org.apache.spark.rdd.RDDBarrier.$anonfun$mapPartitions$2(RDDBarrier.scala:51)
at org.apache.spark.rdd.RDDBarrier.$anonfun$mapPartitions$2$adapted(RDDBarrier.scala:51)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

25/06/13 19:31:13 INFO TaskSetManager: Lost task 0.0 in stage 10.0 (TID 16) on node133, executor 3: ml.dmlc.xgboost4j.java.XGBoostError ([11:31:24] /workspace/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:186: [11:31:24] /workspace/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:89: Check failed: jenv->ExceptionOccurred():
Stack trace:
[bt] (0) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7de4f9ddf58e]
[bt] (1) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(XGBoost4jCallbackDataIterNext+0xb5b) [0x7de4f9dd862b]
[bt] (2) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::IteratorAdapter::Next()+0x1b) [0x7de4fa0c559b]
[bt] (3) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::SimpleDMatrix::SimpleDMatrix >(xgboost::data::IteratorAdapter*, float, int, xgboost::DataSplitMode)+0x2c3) [0x7de4fa23ce73]
[bt] (4) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::DMatrix* xgboost::DMatrix::Create >(xgboost::data::IteratorAdapter*, float, int, std::string const&, xgboost::DataSplitMode)+0x3e) [0x7de4fa1a732e]
[bt] (5) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(XGDMatrixCreateFromDataIter+0x145) [0x7de4f9df01b5]
[bt] (6) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromDataIter+0x229) [0x7de4f9dd4829]
[bt] (7) [0x7de52580f710]

) [duplicate 1]
25/06/13 19:31:13 INFO DAGScheduler: Marking ShuffleMapStage 10 (repartition at XGBoost.scala:289) as failed due to a barrier task failed.
25/06/13 19:31:13 INFO YarnScheduler: Killing all running tasks in stage 10: Task ShuffleMapTask(10, 3) from barrier stage ShuffleMapStage 10 (repartition at XGBoost.scala:289) failed.
25/06/13 19:31:13 INFO DAGScheduler: ShuffleMapStage 10 (repartition at XGBoost.scala:289) failed in 4.996 s due to Stage failed because barrier task ShuffleMapTask(10, 3) finished unsuccessfully.
ml.dmlc.xgboost4j.java.XGBoostError: [11:31:24] /workspace/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:186: [11:31:24] /workspace/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:89: Check failed: jenv->ExceptionOccurred():
Stack trace:
[bt] (0) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7de4f9ddf58e]
[bt] (1) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(XGBoost4jCallbackDataIterNext+0xb5b) [0x7de4f9dd862b]
[bt] (2) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::IteratorAdapter::Next()+0x1b) [0x7de4fa0c559b]
[bt] (3) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::SimpleDMatrix::SimpleDMatrix >(xgboost::data::IteratorAdapter*, float, int, xgboost::DataSplitMode)+0x2c3) [0x7de4fa23ce73]
[bt] (4) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::DMatrix* xgboost::DMatrix::Create >(xgboost::data::IteratorAdapter*, float, int, std::string const&, xgboost::DataSplitMode)+0x3e) [0x7de4fa1a732e]
[bt] (5) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(XGDMatrixCreateFromDataIter+0x145) [0x7de4f9df01b5]
[bt] (6) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromDataIter+0x229) [0x7de4f9dd4829]
[bt] (7) [0x7de52580f710]

25/06/13 19:31:13 INFO DAGScheduler: Resubmitting ShuffleMapStage 10 (repartition at XGBoost.scala:289) due to barrier stage failure.
25/06/13 19:31:13 INFO DAGScheduler: Marking ShuffleMapStage 10 (repartition at XGBoost.scala:289) as failed due to a barrier task failed.
25/06/13 19:31:13 INFO YarnScheduler: Killing all running tasks in stage 10: Task ShuffleMapTask(10, 0) from barrier stage ShuffleMapStage 10 (repartition at XGBoost.scala:289) failed.
25/06/13 19:31:13 INFO DAGScheduler: ShuffleMapStage 10 (repartition at XGBoost.scala:289) failed in 4.998 s due to Stage failed because barrier task ShuffleMapTask(10, 0) finished unsuccessfully.
ml.dmlc.xgboost4j.java.XGBoostError: [11:31:24] /workspace/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:186: [11:31:24] /workspace/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:89: Check failed: jenv->ExceptionOccurred():
Stack trace:
[bt] (0) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7de4f9ddf58e]
[bt] (1) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(XGBoost4jCallbackDataIterNext+0xb5b) [0x7de4f9dd862b]
[bt] (2) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::IteratorAdapter::Next()+0x1b) [0x7de4fa0c559b]
[bt] (3) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::data::SimpleDMatrix::SimpleDMatrix >(xgboost::data::IteratorAdapter*, float, int, xgboost::DataSplitMode)+0x2c3) [0x7de4fa23ce73]
[bt] (4) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(xgboost::DMatrix* xgboost::DMatrix::Create >(xgboost::data::IteratorAdapter*, float, int, std::string const&, xgboost::DataSplitMode)+0x3e) [0x7de4fa1a732e]
[bt] (5) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(XGDMatrixCreateFromDataIter+0x145) [0x7de4f9df01b5]
[bt] (6) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000004/tmp/libxgboost4j2321261085884125079.so(Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromDataIter+0x229) [0x7de4f9dd4829]
[bt] (7) [0x7de52580f710]

25/06/13 19:31:14 INFO DAGScheduler: Resubmitting failed stages
25/06/13 19:31:14 INFO DAGScheduler: Submitting ShuffleMapStage 10 (MapPartitionsRDD[49] at repartition at XGBoost.scala:289), which has no missing parents
25/06/13 19:31:14 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 132.1 KiB, free 3.4 GiB)
25/06/13 19:31:14 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 47.9 KiB, free 3.4 GiB)
25/06/13 19:31:14 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on 192.168.0.134:42999 (size: 47.9 KiB, free: 3.4 GiB)
25/06/13 19:31:14 INFO SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1611
25/06/13 19:31:14 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 10 (MapPartitionsRDD[49] at repartition at XGBoost.scala:289) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
25/06/13 19:31:14 INFO YarnScheduler: Adding task set 10.1 with 4 tasks resource profile 0
25/06/13 19:31:14 INFO FairSchedulableBuilder: Added task set TaskSet_10.1 tasks to pool default
25/06/13 19:31:14 WARN TaskSetManager: Lost task 1.0 in stage 10.0 (TID 17) (node131 executor 8): TaskKilled (Task ShuffleMapTask(10, 0) from barrier stage ShuffleMapStage 10 (repartition at XGBoost.scala:289) failed.)
25/06/13 19:31:14 INFO DAGScheduler: Ignoring task failure from ShuffleMapTask(10, 1) as it's from ShuffleMapStage 10 attempt 0 and there is a more recent attempt for that stage (attempt 1) running
25/06/13 19:31:14 WARN TaskSetManager: Lost task 2.0 in stage 10.0 (TID 18) (node131 executor 6): TaskKilled (Task ShuffleMapTask(10, 0) from barrier stage ShuffleMapStage 10 (repartition at XGBoost.scala:289) failed.)
25/06/13 19:31:14 INFO YarnScheduler: Removed TaskSet 10.0, whose tasks have all completed, from pool default
25/06/13 19:31:14 INFO DAGScheduler: Ignoring task failure from ShuffleMapTask(10, 2) as it's from ShuffleMapStage 10 attempt 0 and there is a more recent attempt for that stage (attempt 1) running
25/06/13 19:31:17 INFO TaskSetManager: Starting task 0.0 in stage 10.1 (TID 20) (node133, executor 3, partition 0, NODE_LOCAL, 9991 bytes)
25/06/13 19:31:17 INFO TaskSetManager: Starting task 1.0 in stage 10.1 (TID 21) (node133, executor 3, partition 1, RACK_LOCAL, 9991 bytes)
25/06/13 19:31:17 INFO TaskSetManager: Starting task 2.0 in stage 10.1 (TID 22) (node131, executor 6, partition 2, RACK_LOCAL, 9991 bytes)
25/06/13 19:31:17 INFO TaskSetManager: Starting task 3.0 in stage 10.1 (TID 23) (node133, executor 3, partition 3, NODE_LOCAL, 9991 bytes)
25/06/13 19:31:17 INFO YarnScheduler: Successfully scheduled all the 4 tasks for barrier stage 10.
25/06/13 19:31:17 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on node131:33239 (size: 47.9 KiB, free: 4.6 GiB)
25/06/13 19:31:17 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on node133:44851 (size: 47.9 KiB, free: 4.6 GiB)
25/06/13 19:31:23 WARN TaskSetManager: Lost task 2.0 in stage 10.1 (TID 22) (node131 executor 6): ml.dmlc.xgboost4j.java.XGBoostError: [11:31:34] /workspace/src/collective/result.cc:78:
- [comm.cc:220|11:31:34]: Failed to bootstrap the communication group.
- [comm.cc:239|11:31:34]: Bootstrap failed.
- [comm.cc:41|11:31:34]: Failed to connect to the tracker.
- [socket.cc:189|11:31:34]: Failed to connect to 192.168.0.134:37183
- [socket.h:79|11:31:34]: Poll error condition:操作现在正在进行 code:115 操作现在正在进行
- [socket.h:348|11:31:34]: Socket error. 连接被拒绝
Stack trace:
[bt] (0) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000009/tmp/libxgboost4j1069947293396215187.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x77fb3e9df58e]
[bt] (1) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000009/tmp/libxgboost4j1069947293396215187.so(xgboost::collective::SafeColl(xgboost::collective::Result const&)+0x7e) [0x77fb3eb8223e]
[bt] (2) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000009/tmp/libxgboost4j1069947293396215187.so(xgboost::collective::RabitComm::RabitComm(std::string const&, int, std::chrono::duration >, int, std::string, xgboost::StringView)+0x4a3) [0x77fb3eb58413]
[bt] (3) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000009/tmp/libxgboost4j1069947293396215187.so(xgboost::collective::CommGroup::Create(xgboost::Json)+0x1d14) [0x77fb3eb662f4]
[bt] (4) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000009/tmp/libxgboost4j1069947293396215187.so(xgboost::collective::GlobalCommGroupInit(xgboost::Json)+0x5c) [0x77fb3eb6830c]
[bt] (5) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000009/tmp/libxgboost4j1069947293396215187.so(XGCommunicatorInit+0x4c) [0x77fb3eb17efc]
[bt] (6) /opt/module/hadoop/data/tmp/nm-local-dir/usercache/zxc/appcache/application_1749544606967_0028/container_1749544606967_0028_01_000009/tmp/libxgboost4j1069947293396215187.so(Java_ml_dmlc_xgboost4j_java_XGBoostJNI_CommunicatorInit+0xf3) [0x77fb3e9d5293]
[bt] (7) [0x77fb6980f710]

at ml.dmlc.xgboost4j.java.Communicator.checkCall(Communicator.java:57)
at ml.dmlc.xgboost4j.java.Communicator.init(Communicator.java:71)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$.$anonfun$train$2(XGBoost.scala:255)
at org.apache.spark.rdd.RDDBarrier.$anonfun$mapPartitions$2(RDDBarrier.scala:51)
at org.apache.spark.rdd.RDDBarrier.$anonfun$mapPartitions$2$adapted(RDDBarrier.scala:51)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

```
code:

```
$SPARK_HOME/bin/spark-submit \
--master yarn \
--deploy-mode client \
--class com.lanhomg.training.ModelTrainingApp \
--name ModelTrainingApp2 \
--num-executors 10 \
--executor-cores 4 \
--executor-memory 8g \
--conf spark.executor.memoryOverhead=1024 \
--conf spark.driver.memory=6g \
--conf spark.scheduler.mode=FAIR \
--conf spark.default.parallelism=300 \
--conf spark.sql.shuffle.partitions=300 \
--conf spark.yarn.submit.waitAppCompletion=false \
--conf spark.driver.bindAddress=0.0.0.0 \
--conf spark.driver.host=192.168.0.134 \
$SPARK_HOME/jobs/model-training-1.0-SNAPSHOT-jar-with-dependencies.jar

```

import ml.dmlc.xgboost4j.scala.spark.{XGBoostRegressionModel, XGBoostRegressor}
import org.apache.spark.ml.feature.{OneHotEncoder, SQLTransformer, StringIndexer, VectorAssembler}
import org.apache.spark.ml.{Pipeline, PipelineStage}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

object XGBoostTrainer {

def train(spark: SparkSession,
dataPath: String,
modelOutputPath: String,
nativeModelPath: String = "",
device: String = "cpu",
numWorkers: Int = 2): Unit = {

try {
val rawData = spark.read.parquet(dataPath)

val dataWithLabel = rawData.withColumn("label", col("conductivity").cast("double"))

val dopantTransformer = new SQLTransformer()
.setStatement(
"""
SELECT *,
CAST(get_json_object(dopant_elements, '$.Y') AS DOUBLE) AS dopant_Y,
CAST(get_json_object(dopant_elements, '$.Sc') AS DOUBLE) AS dopant_Sc,
CAST(get_json_object(dopant_elements, '$.Bi') AS DOUBLE) AS dopant_Bi
FROM __THIS__
"""
)

val categoricalCols = Seq("crystal_phase", "method", "material_source")
val stages = scala.collection.mutable.ArrayBuffer[PipelineStage]()
stages += dopantTransformer

val encodedCols = categoricalCols.map { colName =>
val indexer = new StringIndexer()
.setInputCol(colName)
.setOutputCol(colName + "_idx")
.setHandleInvalid("keep")

val encoder = new OneHotEncoder()
.setInputCol(colName + "_idx")
.setOutputCol(colName + "_vec")

stages += indexer
stages += encoder

colName + "_vec"
}

val featureCols = Seq(
"dopant_Y", "dopant_Sc", "dopant_Bi",
"sintering_temp", "sintering_time", "test_temp"
) ++ encodedCols

val assembler = new VectorAssembler()
.setInputCols(featureCols.toArray)
.setOutputCol("features")
.setHandleInvalid("skip")
stages += assembler

val xgbParams = Map(
"eta" -> 0.1f,
"max_depth" -> 6,
"objective" -> "reg:squarederror",
"num_round" -> 100,
"num_workers" -> numWorkers,
"device" -> device,
"tracker_conf" -> "host=192.168.0.134,port=41931"
)

val xgb = new XGBoostRegressor(xgbParams)
.setFeaturesCol("features")
.setLabelCol("label")

stages += xgb

val pipeline = new Pipeline().setStages(stages.toArray)
val model = pipeline.fit(dataWithLabel)

model.write.overwrite().save(modelOutputPath)

if (nativeModelPath.nonEmpty) {
val xgbModel = model.stages.last.asInstanceOf[XGBoostRegressionModel]
xgbModel.nativeBooster.saveModel(nativeModelPath)
}

} catch {
case e: Exception =>
e.printStackTrace()
}
}
}
```
```
object ModelTrainingApp {
def main(args: Array[String]): Unit = {
val dataPath = "hdfs://node131:9000/user/hive/warehouse/material_conductivity_data1w/"
val modelOutputPath = "hdfs://node131:9000/model/xgbt/pipeline/"
val nativeModel="hdfs://node131:9000/model/xgbt/native/xgbt.model"
val spark = SparkSession.builder()
.appName("model-trainer")
//.master("local[*]")
.getOrCreate()

XGBoostTrainer.train(spark,dataPath,modelOutputPath,nativeModel,numWorkers = 4)

spark.stop()

}
}

```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Failed to connect to the tracker

Description

Contributor guide

Assessment

Get new issues in your inbox