aws / aws/amazon-sagemaker-examples

[Bug Report] Sagemaker debugger issue - CreateXgboostReport processing job fails when training XGBoost image version >= 1.3-1

Open
#3,659 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
[Add the link to the notebook.](https://github.com/aws/amazon-sagemaker-examples/blob/15c4d2433ca1839fc014e01dfeaece5cc87e13b0/sagemaker-debugger/xgboost_training_report/higgs_boson_detection.ipynb)

**Describe the bug**
If you run [Amazon SageMaker Debugger XGBoost training report for Higgs Boson Detection Challenge](https://github.com/aws/amazon-sagemaker-examples/blob/15c4d2433ca1839fc014e01dfeaece5cc87e13b0/sagemaker-debugger/xgboost_training_report/higgs_boson_detection.ipynb) notebook (latest version to date) from the `sagemaker-examples` repo with newer xgboost container versions `1.3-1` or `1.5-1`, then, as a result, you get `CreateXgboostReport` processing job failed.

**To reproduce**
Replace
```python
xgboost_container = image_uris.retrieve("xgboost", region, "1.2-1")
```
line of code with the newer container versions - `1.3-1` or `1.5-1`
```python
xgboost_container = image_uris.retrieve("xgboost", region, "1.3-1") # or 1.5-1
```
and run the whole notebook.

**Logs**
```
[2022-11-07 13:29:11.235 <...> INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
[2022-11-07 13:29:13.678 <...> INFO local_trial.py:35] Loading trial base_trial at path /opt/ml/processing/input/tensors
Exception during rule evaluation: Customer Error: No debugging data was saved by the training job. Check that the debugger hook was configured correctly before starting the training job. Exception: Training job has ended. All the collection files could not be loaded
Traceback (most recent call last):
File "evaluate.py", line 119, in _create_trials
range_steps=(self.start_step, self.end_step))
File "/usr/local/lib/python3.7/site-packages/smdebug/trials/utils.py", line 25, in create_trial
return LocalTrial(name=name, dirname=path, **kwargs)
File "/usr/local/lib/python3.7/site-packages/smdebug/trials/local_trial.py", line 36, in __init__
self._load_collections()
File "/usr/local/lib/python3.7/site-packages/smdebug/trials/trial.py", line 168, in _load_collections
_wait_for_collection_files(1) # wait for the first collection file
File "/usr/local/lib/python3.7/site-packages/smdebug/trials/trial.py", line 165, in _wait_for_collection_files
raise MissingCollectionFiles
smdebug.exceptions.MissingCollectionFiles: Training job has ended. All the collection files could not be loaded
```

I see several possible issue sources, however, I am not sure which one is the *one*:
- XGBoost container images:
- from `1.2-1` up to the latest version to date `1.5-1` code pieces that contain [`smdebug` library](https://github.com/awslabs/sagemaker-debugger) related code have not been changed.
- `smdebug` library:
- although the version of the library listed in [`requirements.txt`](https://github.com/aws/sagemaker-xgboost-container/blob/master/requirements.txt) had changed from `smdebug==1.0.7` in [`1.2-1`](https://github.com/aws/sagemaker-xgboost-container/blob/v1.2-1/requirements.txt#L20) to `smdebug==1.0.10` in [`1.2-2`](https://github.com/aws/sagemaker-xgboost-container/blob/v1.2-2/requirements.txt#L20) [and](https://github.com/aws/sagemaker-xgboost-container/blob/v1.3-1/requirements.txt#L22) [later](https://github.com/aws/sagemaker-xgboost-container/blob/v1.5-1/requirements.txt#L22), running the example notebook with xgboost container version `1.2-2` worked totally fine.
- `972752614525.dkr.ecr.ap-southeast-1.amazonaws.com/sagemaker-debugger-rules:latest` image:
- in all cases, this image version has been used. I doubt that the image had been updated while I was running my "experiments".
- `sagemaker==2.112.2`.

Contributor guide

Open the contributing guide

Research direction

Run the linked higgs_boson_detection.ipynb with XGBoost container versions 1.3-1 and 1.5-1, comparing against 1.2-1 and 1.2-2. Start with the CreateXgboostReport processing job and the reported MissingCollectionFiles traceback, then inspect the referenced smdebug requirements versions and container-related code. Done means the notebook completes successfully with the newer container versions and produces the training report.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.