dmlc / dmlc/xgboost

DataUtils.repartitionRDDs shuffle read can only one task running?

Open
#9,564 3 comments 0 reactions 0 assignees View on GitHub
status: need update
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

hi,when i run xgboost-spark_2.12:1.7.6, when the hdfs train files number = 918 and the shuffle write task number
is 918 but shuffle read only one task running at one time, which cost hours to handle 1 million raws of train data
code: org.apache.spark.rdd.RDD.repartition(RDD.scala:462)
ml.dmlc.xgboost4j.scala.spark.util.DataUtils$.$anonfun$repartitionRDDs$4(DataUtils.scala:96)
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
scala.collection.TraversableLike.map(TraversableLike.scala:238)
scala.collection.TraversableLike.map$(TraversableLike.scala:231)
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
ml.dmlc.xgboost4j.scala.spark.util.DataUtils$.repartitionRDDs(DataUtils.scala:94)
ml.dmlc.xgboost4j.scala.spark.util.DataUtils$.convertDataFrameToXGBLabeledPointRDDs(DataUtils.scala:155)
ml.dmlc.xgboost4j.scala.spark.PreXGBoost$.buildDatasetToRDD(PreXGBoost.scala:156)
ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier.train(XGBoostClassifier.scala:197)
ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier.train(XGBoostClassifier.scala:34)
org.apache.spark.ml.Predictor.fit(Predictor.scala:150)

stage:151 Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write
2023/09/09 22:31:59 33 min 0/1 (1 running) 3.0 GiB

stage:149 Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write
2023/09/09 22:31:58. 3.0 min 918/918 15.5 GiB 10.5 GiB

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.