aws / aws/sagemaker-python-sdk

ModelTrainer API cannot configure the right Session

Open
#5,237 5 comments 0 reactions 0 assignees View on GitHub
component: training type: bug
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the bug**
I'm trying to train a model using the `sagemaker.modules.train.ModelTrainer` API. However, it keeps trying to validate the SageMaker session using Pydantic, only never to accept any possible input. It spits out the Validation Error you see in the screenshot attached below.

**To reproduce**
1. Write a ModelTrainer compatible script
2. Write the following code:

```python
model_trainer = ModelTrainer(
training_image=image_uri, compute=compute, source_code=source_code,
hyperparameters=hyperparameters, environment=env,
base_job_name=job_prefix,
stopping_condition=StoppingCondition(max_runtime_in_seconds=90000),
checkpoint_config=CheckpointConfig(s3_uri=f"{checkpoint_s3_path}/{job_prefix}"),
)
model_trainer.fit(...)
```

**Expected behavior**
Training to start

**Screenshots or logs**

Image

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.248.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A
- **Framework version**: N/A
- **Python version**: 3.12
- **CPU or GPU**: GPU
- **Custom Docker image (Y/N)**: N

**Additional context**
Tried to downgrade to other SageMaker SDK versions, but couldn't get to a working one.

Contributor guide

Open the contributing guide

Research direction

Start with the ModelTrainer API and reproduce the supplied fit(...) example using SageMaker Python SDK 2.248.0 and Python 3.12. Trace the Pydantic validation of the SageMaker session and determine what valid session input is accepted; done means ModelTrainer.fit(...) starts training without a validation error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
api, cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.