aws / aws/amazon-sagemaker-examples
[Bug Report] `introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb` fails during training w/ data load error
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb
introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb
**Describe the bug**
Notebook fails during training step. Inspecting job failure reason gives:
```
'AlgorithmError: framework error: \nTraceback (most recent call last):\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/data_utils.py", line 414, in _get_parquet_dmatrix_pipe_mode\n for record in reader:\nmlio.CorruptHeaderError: The record does not start with the Parquet magic number.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_trainer.py", line 84, in train\n entrypoint()\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/training.py", line 94, in main\n train(framework.training_env())\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/training.py", line 90, in train\n run_algorithm_mode()\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/training.py", line 68, in run_algorithm_mode\n checkpoint_config=checkpoint_config\n File "/miniconda3/lib/python3.7/site-packages/sag'
```
**To reproduce**
Run notebook - after failure inspect job w/ `client.describe_training_job(TrainingJobName=job_name)["FailureReason"]`
**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.
[xgboost_parquet_input_training.pdf](https://github.com/aws/amazon-sagemaker-examples/files/7123398/xgboost_parquet_input_training.pdf)
Contributor guide
Research direction
Start with introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb and run the notebook through its training step. Inspect client.describe_training_job(TrainingJobName=job_name)["FailureReason"] and the attached failure log, focusing on the parquet data-load error. Done means the notebook's training job completes without the CorruptHeaderError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100