Performance Degradation When Upgrading to v1.60
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
I recently tried updating from 0.90 to 1.60. However, my distributed training job (using the approx method) is ~8 times slower now. On version 0.90, each boosting round took about 25 seconds. On version 1.6, each boosting round is now taking around 3 minutes.
Even the hist method on v1.60 is slower than using approx on v0.90.
My dataset has ~5000 features and 500K rows. The exact same parameters and exact same data are being used in my training runs for both versions. The only difference is the version. I cannot share the dataset since it is a work dataset. Roughly 20% of the values in the data are null.
One thing I've noticed, on version 0.90, is that if I increase the nthread parameter the time taken per boosting round goes down. If I decrease the nthread parameter, the time taken per boosting round goes up. This makes sense.
However, on version 1.60, increasing or decreasing the nthread parameter doesn't seem to have any affect. I'm wondering if this is related in some way.
Here's the relevant code snippet:
```
dtrain = get_dmatrix("my_training_data_path.csv", "csv")
dval = get_dmatrix("my_validation_data_path.csv", "csv")
watchlist = [(dtrain, "train"), (dval, "validation")]
hyperparameters = {
"alpha": 0,
"gamma": 0.15,
"learning_rate": 0.1,
"max_depth": 8,
"num_round": 120,
"objective": "binary:logistic",
"scale_pos_weight": 1,
"subsample": 0.8,
"tree_method": "approx",
}
feval = None
maximize = True
progress_metrics = {}
booster = xgboost.train(
params=hyperparameters,
feval=feval,
dtrain=dtrain,
evals=watchlist,
evals_result=progress_metrics,
maximize=maximize,
num_boost_round=num_boost_round,
early_stopping_rounds=num_early_stopping_rounds,
callbacks=callbacks,
)
```
Using 2 ml.m5.12xlarge (48vCPU's, 192 GiB RAM) on AWS SageMaker for each training job. Python 3.7.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.