[xgboost4j-spark]process_type params does not exist in release1.0.0?
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
### 一、abount process_type
1. get this param in https://github.com/dmlc/xgboost/blob/release_1.0.0/doc/parameter.rst
2. set `process_type=update` and run , result:spark logs
```
23/08/28 16:21:17 INFO XGBoostSpark: Running XGBoost 1.0.0 with parameters:
alpha -> 0.002
min_child_weight -> 1.0
sample_type -> uniform
base_score -> 0.5
rabit_timeout -> -1
colsample_bylevel -> 0.6
grow_policy -> depthwise
skip_drop -> 0.0
lambda_bias -> 0.0
silent -> 0
scale_pos_weight -> 1.0
seed -> 0
cache_training_set -> false
features_col -> features
num_early_stopping_rounds -> 0
label_col -> label
num_workers -> 150
subsample -> 1.0
lambda -> 0.01
max_depth -> 4
probability_col -> probability
raw_prediction_col -> rawPrediction
tree_limit -> 0
custom_eval -> null
dmlc_worker_connect_retry -> 5
rate_drop -> 0.0
max_bin -> 16
train_test_ratio -> 1.0
use_external_memory -> true
objective -> binary:logistic
eval_metric -> auc
num_round -> 10
timeout_request_workers -> 1800000
missing -> NaN
rabit_ring_reduce_threshold -> 32768
checkpoint_path ->
tracker_conf -> TrackerConf(0,python)
tree_method -> auto
max_delta_step -> 0.0
eta -> 0.15
verbosity -> 1
colsample_bytree -> 0.6
normalize_type -> tree
allow_non_zero_for_missing -> true
custom_obj -> null
gamma -> 0.03
sketch_eps -> 0.03
nthread -> 1
prediction_col -> prediction
checkpoint_interval -> -1
```
3. check source code BoostreParams.scala,but can't find it。
### 二、about one billion sample data xgboost train
1. data:one or two billion, vector size about 2000(in fact more than)
2. dataFrame too big to cache in memory,so cache in disk storage,but train time too long。so far,i try 500 million datas two times。
```
# 1.2T 5hour19min total executor cores:150C
23/08/25 11:41:40 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 11:41:40,986 INFO @tracker All of 150 nodes getting started
23/08/25 12:53:48 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 12:53:48,012 INFO [0] train-auc:0.734405
23/08/25 13:27:49 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 13:27:49,444 INFO [1] train-auc:0.739572
23/08/25 13:47:27 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 13:47:27,631 INFO [2] train-auc:0.739964
23/08/25 14:13:54 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 14:13:54,872 INFO [3] train-auc:0.741611
23/08/25 14:41:50 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 14:41:50,012 INFO [4] train-auc:0.747264
23/08/25 15:03:43 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 15:03:43,079 INFO [5] train-auc:0.747870
23/08/25 15:34:01 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 15:34:01,426 INFO [6] train-auc:0.755129
23/08/25 15:58:52 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 15:58:52,605 INFO [7] train-auc:0.757447
23/08/25 16:23:16 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 16:23:16,961 INFO [8] train-auc:0.760328
23/08/25 16:43:02 INFO RabitTracker$TrackerProcessLogger: 2023-08-25 16:43:02,451 INFO [9] train-auc:0.760338
# 1.2T 3hour14min total executor cores:150C
23/08/28 17:15:03 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 17:15:03,372 INFO [0] train-auc:0.734405
23/08/28 17:34:30 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 17:34:30,593 INFO [1] train-auc:0.739572
23/08/28 17:54:04 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 17:54:04,159 INFO [2] train-auc:0.739964
23/08/28 18:12:24 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 18:12:24,665 INFO [3] train-auc:0.741611
23/08/28 18:21:29 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 18:21:29,123 INFO [4] train-auc:0.747264
23/08/28 18:40:59 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 18:40:59,078 INFO [5] train-auc:0.747870
23/08/28 18:49:56 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 18:49:56,984 INFO [6] train-auc:0.755129
23/08/28 19:05:39 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 19:05:39,836 INFO [7] train-auc:0.757447
23/08/28 19:13:58 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 19:13:58,568 INFO [8] train-auc:0.760328
23/08/28 19:21:39 INFO RabitTracker$TrackerProcessLogger: 2023-08-28 19:21:39,710 INFO [9] train-auc:0.760338
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.