aws / aws/sagemaker-python-sdk
[Bug] ModelTrainer with no input channels emits InputDataConfig: [], which CreatePipeline rejects (min=1) — v2 omitted the key
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 32
Description
**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
**Describe the bug**
A `TrainingStep` built from a `ModelTrainer` that has no input channels serializes `"InputDataConfig": []` into the pipeline definition. `CreatePipeline` rejects the whole definition:
```
botocore.exceptions.ClientError: An error occurred (ValidationException) when
calling the CreatePipeline operation: Unable to parse pipeline definition.
Model Validation failed: Length of container InputDataConfig=0 cannot be less
than min=1.
```
The SageMaker API accepts an **absent** `InputDataConfig` — it is not among `CreateTrainingJob`'s required members — but rejects an **empty** one, because botocore's own service model gives that member `min=1`. It is the only `min=1` list in the `CreateTrainingJob` shape.
Under the v2 SDK, `TrainingStep(name=..., estimator=...)` with no `inputs=` omitted the key entirely and the same pipeline created successfully. So this is a v2 → v3 behaviour regression for any training job whose data does not arrive over an S3 channel — in our case a fine-tuning job that pulls its dataset, base model and resume checkpoint from the HuggingFace Hub inside the container.
Cause — `sagemaker-train`, `src/sagemaker/train/model_trainer.py`:
```python
# :583
final_input_data_config = self.input_data_config.copy() if self.input_data_config else []
...
# :739
"input_data_config": final_input_data_config,
```
The `else []` makes "no channels" indistinguishable from "an empty list of channels" downstream, and the empty list is serialized rather than dropped. `input_data_config=None` — the default, and what the reproduction passes — takes that branch.
Suggested fix: omit the key when there are no channels, e.g. build the request without `input_data_config` when `final_input_data_config` is falsy. `None` would also work if the serializer drops `None` members.
**To reproduce**
Standalone, no AWS calls, no credentials — the two `mock.patch` calls only stop the SDK reaching IAM/S3 during construction:
```python
import json, os
os.environ.setdefault("AWS_DEFAULT_REGION", "us-west-2")
from unittest import mock
import sagemaker.train.defaults as td
from sagemaker.core.workflow.pipeline_context import PipelineSession
from sagemaker.mlops.workflow.pipeline import Pipeline
from sagemaker.mlops.workflow.steps import TrainingStep
from sagemaker.train import ModelTrainer
from sagemaker.train.configs import Compute
ROLE = "arn:aws:iam::000000000000:role/example"
with mock.patch.object(td, "resolve_and_validate_role",
lambda provided_role=None, **kw: provided_role or ROLE), \
mock.patch.object(PipelineSession, "default_bucket", lambda self: "example-bucket"):
session = PipelineSession()
trainer = ModelTrainer( # no input_data_config: none needed
sagemaker_session=session, role=ROLE, base_job_name="repro",
training_image="000000000000.dkr.ecr.us-west-2.amazonaws.com/example:latest",
compute=Compute(instance_type="ml.m5.large", instance_count=1),
)
step = TrainingStep(name="NoChannels", step_args=trainer.train(wait=False))
definition = json.loads(
Pipeline(name="repro", steps=[step], sagemaker_session=session).definition())
args = definition["Steps"][0]["Arguments"]
print("InputDataConfig present:", "InputDataConfig" in args)
print("InputDataConfig value :", json.dumps(args.get("InputDataConfig")))
```
Output:
```
InputDataConfig present: True
InputDataConfig value : []
```
Calling `pipeline.upsert(role_arn=...)` on that definition raises the `ValidationException` quoted above.
**Expected behavior**
With no input channels, `InputDataConfig` is omitted from the serialized definition, matching v2 and matching what the API accepts.
**Screenshots or logs**
See the `ValidationException` and reproduction output above.
**System information**
- **SageMaker Python SDK version**: sagemaker 3.18.0 (PyPI latest at time of writing); sagemaker-core 2.18.0; sagemaker-train 1.18.0; sagemaker-mlops 1.18.0; sagemaker-serve 1.18.0; boto3/botocore 1.43.53
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: custom training image (data pulled from HuggingFace Hub inside the container)
- **Framework version**: N/A
- **Python version**: 3.12
- **CPU or GPU**: N/A — bug is SDK-side serialization, no job runs
- **Custom Docker image (Y/N)**: Y
**Additional context**
`master` carries the byte-identical `else []`, so this is not fixed in an unreleased commit. Pinning back to the 3.11.0 family avoids it, but that reverts a deliberate change and alters other parts of the definition.
Note for anyone reproducing: `sagemaker.__version__` no longer exists on v3 (the `sagemaker` 3.x wheel is a namespace shim), so version-report snippets that read it raise `AttributeError`.
Workaround we are using — subclass `TrainingStep` and drop the empty container at the point the request first exists as a plain dict (`arguments` is an abstract member of the SDK's own `Step` ABC, so it is the documented seam):
```python
from sagemaker.mlops.workflow.steps import TrainingStep as _SdkTrainingStep
class TrainingStep(_SdkTrainingStep):
@property
def arguments(self) -> dict:
request = super().arguments
if not request.get("InputDataConfig"):
request.pop("InputDataConfig", None)
return request
```
Guarding on emptiness rather than popping unconditionally means a step that _does_ take a channel is unaffected.
Contributor guide
Assessment
This issue has not been assessed yet.