dmlc / dmlc/xgboost

Example of using the XGBoost CV with custom function (Python)

Open
#10,838 0 comments 0 reactions 0 assignees View on GitHub
cross-validation
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Hello XGBoost team,

I am trying to use the XGBoost CV module with Optuna for parameter optimization with the following code:

``` python
def kappa_scorer(y_true, y_pred):
"""Kappa scorer for the XGBoost model."""
return "kappa", cohen_kappa_score(y_true, y_pred)

def objective_xgboost(trial, study_name, X_train, y_train, label_to_idx, exp_type: str):
"""Objective function for the XGBoost classifier with final eval metric as Kappa."""
params = {
"verbosity": 0,
"objective": "multi:softmax",
"num_class": len(label_to_idx),
"feval": kappa_scorer,
"n_estimators": 1000,
"eta": trial.suggest_float("learning_rate", 1e-2, 0.1, log=True),
"max_depth": trial.suggest_int("max_depth", 2, 10),
"colsample_bytree": trial.suggest_float(
"colsample_bytree", 0.1, 0.7
), # Percentage of features used per tree.
"disable_default_eval_metric": 1,
}

# Training data
y_train = y_train.map(label_to_idx)

dtrain = xgb.DMatrix(X_train, label=y_train)

# Optimization of kappa scoe
pruning_callback = optuna.integration.XGBoostPruningCallback(trial, "test-kappa")
xgboost_model = xgb.cv(
params, dtrain, callbacks=[pruning_callback], seed=SEED, nfold=5
)

# Save model
trial_path = _generate_dirs(exp_type)
save_xgboost_trial(
trial=trial, model=xgboost_model, study_name=study_name, trial_dir=trial_path
)

mean_kappa = xgboost_model["test-kappa-mean"].values[-1] # Optimized for kappa

return mean_kappa
```
However, I am not able to extract the metric results from the trained model and stable upon an error: `KeyError: 'test-kappa'`. Is there any pointer to what I am doing wrong?

Thank you.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the example at the xgb.cv entry point with the shown DMatrix, custom scorer, and XGBoostPruningCallback. Trace the evaluation metric names returned by cv and used by the callback; done means the reported KeyError is explained and the supported metric access pattern or documentation example is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.