aws / aws/amazon-sagemaker-examples

[Bug Report] `introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb` fails during training w/ data load error

Open
#2,921 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb

introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb

**Describe the bug**
Notebook fails during training step. Inspecting job failure reason gives:
```
'AlgorithmError: framework error: \nTraceback (most recent call last):\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/data_utils.py", line 414, in _get_parquet_dmatrix_pipe_mode\n for record in reader:\nmlio.CorruptHeaderError: The record does not start with the Parquet magic number.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_trainer.py", line 84, in train\n entrypoint()\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/training.py", line 94, in main\n train(framework.training_env())\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/training.py", line 90, in train\n run_algorithm_mode()\n File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/training.py", line 68, in run_algorithm_mode\n checkpoint_config=checkpoint_config\n File "/miniconda3/lib/python3.7/site-packages/sag'
```

**To reproduce**
Run notebook - after failure inspect job w/ `client.describe_training_job(TrainingJobName=job_name)["FailureReason"]`

**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.
[xgboost_parquet_input_training.pdf](https://github.com/aws/amazon-sagemaker-examples/files/7123398/xgboost_parquet_input_training.pdf)

Contributor guide

Open the contributing guide

Research direction

Start with introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb and run the notebook through its training step. Inspect client.describe_training_job(TrainingJobName=job_name)["FailureReason"] and the attached failure log, focusing on the parquet data-load error. Done means the notebook's training job completes without the CorruptHeaderError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.