aws / aws/amazon-sagemaker-examples
Hyperparameter optimisation XGB model via CV via script mode
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
Hi all,
I followed the script mode example [here](https://github.com/aws/amazon-sagemaker-examples/blob/c391082711d4297d157e3a546be29ef422a5974b/sagemaker-script-mode/sagemaker-script-mode.ipynb
) , specificaly [this](https://github.com/aws/amazon-sagemaker-examples/blob/c391082711d4297d157e3a546be29ef422a5974b/sagemaker-script-mode/xgboost_script/train_deploy_xgboost_with_dependencies.py). This is great and the cross_validation function [here](https://github.com/aws/amazon-sagemaker-examples/blob/c391082711d4297d157e3a546be29ef422a5974b/sagemaker-script-mode/my_custom_library/cross_validation_xgboost.py) returns errors and the model:
```
def cross_validation(df, K, hyperparameters):
"""
Perform cross validation on a dataset.
:param df: pandas.DataFrame
:param K: int
:param hyperparameters: dict
"""
train_indices = list(df.sample(frac=1).index)
k_folds = np.array_split(train_indices, K)
if K == 1:
K = 2
rmse_list = []
for i in range(len(k_folds)):
training_folds = [fold for j, fold in enumerate(k_folds) if j != i]
training_indices = np.concatenate(training_folds)
x_train, y_train = df.iloc[training_indices, 1:], df.iloc[training_indices, :1]
x_validation, y_validation = df.iloc[k_folds[i], 1:], df.iloc[k_folds[i], :1]
dtrain = xgb.DMatrix(data=x_train, label=y_train)
dvalidation = xgb.DMatrix(data=x_validation, label=y_validation)
model = xgb.train(
params=hyperparameters,
dtrain=dtrain,
evals=[(dtrain, "train"), (dvalidation, "validation")],
)
eval_results = model.eval(dvalidation)
rmse_list.append(float(eval_results.split("eval-rmse:")[1]))
return rmse_list, model
```
I wonder if this could be used in Bayesian hyper parameter optimisation in Sagemaker, which I comfortably do but not via CV along [these lines](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-ex-tuning-job.html). So the question is, can a script mode implementation like the above invoked in the "usual" Bayesian HPO framework of Sagemaker. Thanks.
Contributor guide
Assessment
This issue has not been assessed yet.