aws / aws/amazon-sagemaker-examples

Hyperparameter optimisation XGB model via CV via script mode

Open
#3,441 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Hi all,

I followed the script mode example [here](https://github.com/aws/amazon-sagemaker-examples/blob/c391082711d4297d157e3a546be29ef422a5974b/sagemaker-script-mode/sagemaker-script-mode.ipynb
) , specificaly [this](https://github.com/aws/amazon-sagemaker-examples/blob/c391082711d4297d157e3a546be29ef422a5974b/sagemaker-script-mode/xgboost_script/train_deploy_xgboost_with_dependencies.py). This is great and the cross_validation function [here](https://github.com/aws/amazon-sagemaker-examples/blob/c391082711d4297d157e3a546be29ef422a5974b/sagemaker-script-mode/my_custom_library/cross_validation_xgboost.py) returns errors and the model:

```
def cross_validation(df, K, hyperparameters):
"""
Perform cross validation on a dataset.
:param df: pandas.DataFrame
:param K: int
:param hyperparameters: dict
"""
train_indices = list(df.sample(frac=1).index)
k_folds = np.array_split(train_indices, K)
if K == 1:
K = 2

rmse_list = []
for i in range(len(k_folds)):
training_folds = [fold for j, fold in enumerate(k_folds) if j != i]
training_indices = np.concatenate(training_folds)
x_train, y_train = df.iloc[training_indices, 1:], df.iloc[training_indices, :1]
x_validation, y_validation = df.iloc[k_folds[i], 1:], df.iloc[k_folds[i], :1]
dtrain = xgb.DMatrix(data=x_train, label=y_train)
dvalidation = xgb.DMatrix(data=x_validation, label=y_validation)

model = xgb.train(
params=hyperparameters,
dtrain=dtrain,
evals=[(dtrain, "train"), (dvalidation, "validation")],
)
eval_results = model.eval(dvalidation)
rmse_list.append(float(eval_results.split("eval-rmse:")[1]))
return rmse_list, model

```
I wonder if this could be used in Bayesian hyper parameter optimisation in Sagemaker, which I comfortably do but not via CV along [these lines](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-ex-tuning-job.html). So the question is, can a script mode implementation like the above invoked in the "usual" Bayesian HPO framework of Sagemaker. Thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.