aws / aws/sagemaker-python-sdk

[Bug] ModelTrainer with no input channels emits InputDataConfig: [], which CreatePipeline rejects (min=1) — v2 omitted the key

Đang mở Phù hợp với người mới
#6,156 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
2.3k
Fork
1.3k
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
35

Mô tả

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

**Describe the bug**
A `TrainingStep` built from a `ModelTrainer` that has no input channels serializes `"InputDataConfig": []` into the pipeline definition. `CreatePipeline` rejects the whole definition:

```
botocore.exceptions.ClientError: An error occurred (ValidationException) when
calling the CreatePipeline operation: Unable to parse pipeline definition.
Model Validation failed: Length of container InputDataConfig=0 cannot be less
than min=1.
```

The SageMaker API accepts an **absent** `InputDataConfig` — it is not among `CreateTrainingJob`'s required members — but rejects an **empty** one, because botocore's own service model gives that member `min=1`. It is the only `min=1` list in the `CreateTrainingJob` shape.

Under the v2 SDK, `TrainingStep(name=..., estimator=...)` with no `inputs=` omitted the key entirely and the same pipeline created successfully. So this is a v2 → v3 behaviour regression for any training job whose data does not arrive over an S3 channel — in our case a fine-tuning job that pulls its dataset, base model and resume checkpoint from the HuggingFace Hub inside the container.

Cause — `sagemaker-train`, `src/sagemaker/train/model_trainer.py`:

```python
# :583
final_input_data_config = self.input_data_config.copy() if self.input_data_config else []
...
# :739
"input_data_config": final_input_data_config,
```

The `else []` makes "no channels" indistinguishable from "an empty list of channels" downstream, and the empty list is serialized rather than dropped. `input_data_config=None` — the default, and what the reproduction passes — takes that branch.

Suggested fix: omit the key when there are no channels, e.g. build the request without `input_data_config` when `final_input_data_config` is falsy. `None` would also work if the serializer drops `None` members.

**To reproduce**
Standalone, no AWS calls, no credentials — the two `mock.patch` calls only stop the SDK reaching IAM/S3 during construction:

```python
import json, os
os.environ.setdefault("AWS_DEFAULT_REGION", "us-west-2")
from unittest import mock
import sagemaker.train.defaults as td
from sagemaker.core.workflow.pipeline_context import PipelineSession
from sagemaker.mlops.workflow.pipeline import Pipeline
from sagemaker.mlops.workflow.steps import TrainingStep
from sagemaker.train import ModelTrainer
from sagemaker.train.configs import Compute

ROLE = "arn:aws:iam::000000000000:role/example"
with mock.patch.object(td, "resolve_and_validate_role",
lambda provided_role=None, **kw: provided_role or ROLE), \
mock.patch.object(PipelineSession, "default_bucket", lambda self: "example-bucket"):
session = PipelineSession()
trainer = ModelTrainer( # no input_data_config: none needed
sagemaker_session=session, role=ROLE, base_job_name="repro",
training_image="000000000000.dkr.ecr.us-west-2.amazonaws.com/example:latest",
compute=Compute(instance_type="ml.m5.large", instance_count=1),
)
step = TrainingStep(name="NoChannels", step_args=trainer.train(wait=False))
definition = json.loads(
Pipeline(name="repro", steps=[step], sagemaker_session=session).definition())

args = definition["Steps"][0]["Arguments"]
print("InputDataConfig present:", "InputDataConfig" in args)
print("InputDataConfig value :", json.dumps(args.get("InputDataConfig")))
```

Output:

```
InputDataConfig present: True
InputDataConfig value : []
```

Calling `pipeline.upsert(role_arn=...)` on that definition raises the `ValidationException` quoted above.

**Expected behavior**
With no input channels, `InputDataConfig` is omitted from the serialized definition, matching v2 and matching what the API accepts.

**Screenshots or logs**
See the `ValidationException` and reproduction output above.

**System information**
- **SageMaker Python SDK version**: sagemaker 3.18.0 (PyPI latest at time of writing); sagemaker-core 2.18.0; sagemaker-train 1.18.0; sagemaker-mlops 1.18.0; sagemaker-serve 1.18.0; boto3/botocore 1.43.53
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: custom training image (data pulled from HuggingFace Hub inside the container)
- **Framework version**: N/A
- **Python version**: 3.12
- **CPU or GPU**: N/A — bug is SDK-side serialization, no job runs
- **Custom Docker image (Y/N)**: Y

**Additional context**
`master` carries the byte-identical `else []`, so this is not fixed in an unreleased commit. Pinning back to the 3.11.0 family avoids it, but that reverts a deliberate change and alters other parts of the definition.

Note for anyone reproducing: `sagemaker.__version__` no longer exists on v3 (the `sagemaker` 3.x wheel is a namespace shim), so version-report snippets that read it raise `AttributeError`.

Workaround we are using — subclass `TrainingStep` and drop the empty container at the point the request first exists as a plain dict (`arguments` is an abstract member of the SDK's own `Step` ABC, so it is the documented seam):

```python
from sagemaker.mlops.workflow.steps import TrainingStep as _SdkTrainingStep

class TrainingStep(_SdkTrainingStep):
@property
def arguments(self) -> dict:
request = super().arguments
if not request.get("InputDataConfig"):
request.pop("InputDataConfig", None)
return request
```

Guarding on emptiness rather than popping unconditionally means a step that _does_ take a channel is unaffected.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu trong src/sagemaker/train/model_trainer.py tại logic final_input_data_config quanh các dòng 583 và 739, sau đó tái hiện quá trình tuần tự hóa TrainingStep mà không gọi AWS. Xác nhận rằng trainer không có kênh đầu vào sẽ bỏ qua InputDataConfig, trong khi trainer có các kênh vẫn không thay đổi, và xác minh rằng định nghĩa pipeline thu được được CreatePipeline chấp nhận.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
aws, python
Lĩnh vực
cloud, machine-learning
Loại issue
Lỗi
Độ khó
2/5
Thời gian dự kiến
1-3 giờ
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
76/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.