DataUtils.repartitionRDDs shuffle read can only one task running?
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
hi,when i run xgboost-spark_2.12:1.7.6, when the hdfs train files number = 918 and the shuffle write task number
is 918 but shuffle read only one task running at one time, which cost hours to handle 1 million raws of train data
code: org.apache.spark.rdd.RDD.repartition(RDD.scala:462)
ml.dmlc.xgboost4j.scala.spark.util.DataUtils$.$anonfun$repartitionRDDs$4(DataUtils.scala:96)
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
scala.collection.TraversableLike.map(TraversableLike.scala:238)
scala.collection.TraversableLike.map$(TraversableLike.scala:231)
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
ml.dmlc.xgboost4j.scala.spark.util.DataUtils$.repartitionRDDs(DataUtils.scala:94)
ml.dmlc.xgboost4j.scala.spark.util.DataUtils$.convertDataFrameToXGBLabeledPointRDDs(DataUtils.scala:155)
ml.dmlc.xgboost4j.scala.spark.PreXGBoost$.buildDatasetToRDD(PreXGBoost.scala:156)
ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier.train(XGBoostClassifier.scala:197)
ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier.train(XGBoostClassifier.scala:34)
org.apache.spark.ml.Predictor.fit(Predictor.scala:150)
stage:151 Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write
2023/09/09 22:31:59 33 min 0/1 (1 running) 3.0 GiB
stage:149 Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write
2023/09/09 22:31:58. 3.0 min 918/918 15.5 GiB 10.5 GiB
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.