aws / aws/amazon-sagemaker-examples

Notebook xgboost_customer_churn fails with XGBoost 1.5-1 and 1.3-1 Images

Open
#3,578 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
[xgboost_customer_churn](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn.ipynb)

**Describe the bug**
Notebook [xgboost_customer_churn](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn.ipynb) fails during training, specifically in cell starting with line of code `sess = sagemaker.Session()`. Looks like coding incompatibilities with the latest version(s).

Fails with v1.5-1 and v1.3-1 - same error (see below). Running with the older v1.2-1 gets through this issue, but may result in other issues.

**To reproduce**
Run the notebook to the training cell which fails, starting with `sess = sagemaker.Session()`.

**Logs**
If applicable, add logs to help explain your problem.
```text
[2022-08-2S 15:23:59.038 ip-10-0-198-79.ec2.internal:1 INFO hook.py:200] tensorboard_dir has not been set for the hook. SNOeb ug will not be exporting tensorboard summaries.
[2022-08-25 15:23:59.031 ip 10 8 198 79.ec2.internal:1 INFO profiler_config_parser.py:102] User has disabled profiler.
[2022-08-25 15:23:59.032 ip-10-0-198-79.ec2.internal:1 INFO hook.py:255] Saving to /opt/ml/output/tensors
[2022-88-25 15:23:59.032 ip-10-0-198-79.ec2.internal:1 INFO state_store.py:77) The checkpoint config file /opt/ml/input/confi g/checkpointconfig.json does not exist. (2022-08-25:15:23:59:INF0) Debug hook created from config
[2022-88-25:15:23:59:ERROR] Reporting training FAILURE
[2022-08-25:15:23:59:ERROR] framework error: Traceback (most recent call last): File -/miniconda3/lib/python3.8/site-packages/sagemaker_xgboost_container/algorithm_mode/train.py", line 238, in trainjob bst xgb.train(train_cfg, train_dmatrix, num_boost_round-num_round-iteration, eyals-watchlist, File "/miniconda3/11b/python3.8/site-packages/xgboost/training.py", line 188, in train bst - _train_internal(params, dtrain, File -/miniconda3/lib/python3.8/site-packages/xgboost/training.py-, line 61, in _train_internal assert all(isinstance(c, callback.Trainingallback) AssertionError: You can't mix new and old callback styles. During handling of the above exception, another exception occurred:
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.