dmlc / dmlc/xgboost

In spark.XGBoost.trainDistributed, the parameter n_thread, does not seem to take effect most of the time and still only utilizes a single thread

Open
#9,607 15 comments 0 reactions 0 assignees View on GitHub
status: need update
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

hello,i got a problem when i run xgboost distribute train in yarn. my parameter :
set executor-memory = "32G"
set executor-cores = "8"
set spark.task.cpus = 8
set spark.driver.maxResultSize="24G"

xgboost train param:
set and objective = "binary:logistic"
and num_round = "700"
and missing = "-999.0"
and max_depth= "3"
and num_workers = '1'
and num_early_stopping_rounds = "10"
and n_thread = 8
when task come to this work: **it always 1 running thread, why?** hope for some answers and advice to speed up the trainning
[20:24:42] task 0 got new rank 0
[20:31:41] INFO: /workspace/src/gbm/gbtree.cc:140: Tree method is automatically selected to be 'approx' for distributed training.
[20:31:41] DEBUG: /workspace/src/gbm/gbtree.cc:157: Using tree method: 1
[20:31:41] DEBUG: /workspace/src/gbm/gbtree.cc:122: Using updaters: grow_histmaker
[20:31:41] ======== Monitor (0): TreePruner ========
[20:31:41] INFO: /workspace/src/data/simple_dmatrix.cc:103: Generating new Gradient Index.
[20:35:03] ======== Monitor (0): SortedSketchContainer ========
[20:35:03] AllReduce: 0.017769s, 1 calls @ 17769us

[20:35:03] MakeCuts: 0.02635s, 1 calls @ 26350us

[20:35:03] PushColPage: 22.5197s, 1 calls @ 22519734us

[20:37:06] INFO: /workspace/src/data/simple_dmatrix.cc:103: Generating new Gradient Index.
[20:37:54] ======== Monitor (0): SortedSketchContainer ========
[20:37:54] AllReduce: 0.008843s, 1 calls @ 8843us

[20:37:54] MakeCuts: 0.017436s, 1 calls @ 17436us

[20:37:54] PushColPage: 46.152s, 1 calls @ 46151982us

[20:40:06] INFO: /workspace/src/data/simple_dmatrix.cc:103: Generating new Gradient Index

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.