aws / aws/sagemaker-python-sdk

ModelTrainer API cannot configure the right Session

Open
#5,237 5 comments 0 reactions 0 assignees View on GitHub
component: training type: bug
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the bug**
I'm trying to train a model using the `sagemaker.modules.train.ModelTrainer` API. However, it keeps trying to validate the SageMaker session using Pydantic, only never to accept any possible input. It spits out the Validation Error you see in the screenshot attached below.

**To reproduce**
1. Write a ModelTrainer compatible script
2. Write the following code:

```python
model_trainer = ModelTrainer(
training_image=image_uri, compute=compute, source_code=source_code,
hyperparameters=hyperparameters, environment=env,
base_job_name=job_prefix,
stopping_condition=StoppingCondition(max_runtime_in_seconds=90000),
checkpoint_config=CheckpointConfig(s3_uri=f"{checkpoint_s3_path}/{job_prefix}"),
)
model_trainer.fit(...)
```

**Expected behavior**
Training to start

**Screenshots or logs**

Image

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.248.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A
- **Framework version**: N/A
- **Python version**: 3.12
- **CPU or GPU**: GPU
- **Custom Docker image (Y/N)**: N

**Additional context**
Tried to downgrade to other SageMaker SDK versions, but couldn't get to a working one.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.