aws / aws/amazon-sagemaker-examples

Issues in Training Module

Open
#4,555 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I am facing the following error in the training part. Can someone help me with this?

UnexpectedStatusException: Error for Training job demo-xgboost-(name)-2024-02-05-11-57-23-767: Failed. Reason: AlgorithmError: framework error:
Traceback (most recent call last):
File "/miniconda3/lib/python3.6/site-packages/sagemaker_containers/_trainer.py", line 84, in train
entrypoint()
File "/miniconda3/lib/python3.6/site-packages/sagemaker_xgboost_container/training.py", line 94, in main
train(framework.training_env())
File "/miniconda3/lib/python3.6/site-packages/sagemaker_xgboost_container/training.py", line 90, in train
run_algorithm_mode()
File "/miniconda3/lib/python3.6/site-packages/sagemaker_xgboost_container/training.py", line 68, in run_algorithm_mode
checkpoint_config=checkpoint_config
File "/miniconda3/lib/python3.6/site-packages/sagemaker_xgboost_container/algorithm_mode/train.py", line 105, in sagemaker_train
train_dmatrix, val_dmatrix = get_validated_dmatrices(train_path, val_path, file_type, csv_weights, is_pipe)
File "/miniconda3/lib/python3.6/site-packages/sagemaker_xgboost_container/algorithm_mode/train.py", line 61, in get_validated_dmatrices
if train_files_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported training job and inspect the traceback through sagemaker_xgboost_container/training.py and algorithm_mode/train.py, especially get_validated_dmatrices. The issue body is truncated before the failure details, so first identify the complete error and the training notebook or entry point involved; done means the training module completes successfully or the failure is documented with a clear fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.