aws / aws/sagemaker-python-sdk

`_deploy_for_ic` passes `instance_type` twice to `_deploy()` causing `TypeError` when deploying `CustomOrchestrator` as Inference Component

Đang mở Phù hợp với người mới
#6,199 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
component: model builder type: bug
Ngôn ngữ chính
Python
Star
2.3k
Fork
1.3k
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
35

Mô tả

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

**Describe the bug**

When deploying a `CustomOrchestrator` as an Inference Component — as documented in the [Build and deploy AI inference workflows with new enhancements to the Amazon SageMaker Python SDK](https://aws.amazon.com/blogs/machine-learning/build-and-deploy-ai-inference-workflows-with-new-enhancements-to-the-amazon-sagemaker-python-sdk/) blog post and the [Llama3.1-Mistral reference notebook](https://github.com/aws-samples/sagemaker-genai-hosting-examples/blob/main/Llama3.1-Mistral-workflow/Llama3.1-8B-Mistral-7B-inference-orchestrator.ipynb) — calling `deploy()` raises a `TypeError: got multiple values for keyword argument 'instance_type'`.

The issue is in `_deploy_for_ic()` ([model_builder.py L4227-4237](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L4227)): `instance_type` and `initial_instance_count` are passed **both** as explicit keyword arguments **and** via `**kwargs` spread to `self._deploy()`.

**To reproduce**

```python
from sagemaker.serve.model_builder import ModelBuilder, SchemaBuilder
from sagemaker.serve.spec.inference_base import CustomOrchestrator
from sagemaker.core.inference_config import ResourceRequirements
from sagemaker.core.helper.session_helper import Session, get_execution_role

class MyOrchestrator(CustomOrchestrator):
def __init__(self, endpoint_name, component_names):
super().__init__()
self.endpoint_name = endpoint_name
self.component_names = component_names

def handle(self, data, context=None):
import json
response = self.client.invoke_endpoint(
EndpointName=self.endpoint_name,
InferenceComponentName=self.component_names[0],
Body=data if isinstance(data, (str, bytes)) else json.dumps(data),
ContentType="application/json"
)
return json.loads(response["Body"].read())

role = get_execution_role()
sess = Session()

# Step 1: Build the orchestrator
orchestrator = ModelBuilder(
inference_spec=MyOrchestrator(
endpoint_name="my-existing-endpoint",
component_names=["base-ic", "adapter-ic"],
),
dependencies={"auto": False, "custom": ["cloudpickle"]},
sagemaker_session=sess,
role_arn=role,
schema_builder=SchemaBuilder(sample_input="Test", sample_output={"generated_text": "test"}),
)

# Workaround for missing constructor fields (separate issue)
orchestrator.resource_requirements = ResourceRequirements(
requests={"memory": 4096, "num_accelerators": 1, "copies": 1, "num_cpus": 2}
)
orchestrator.inference_component_name = "my-orchestrator-ic"

orchestrator.build()

# Step 2: Deploy — this triggers the bug
orchestrator.deploy(
endpoint_name="my-existing-endpoint",
custom_orchestrator_instance_type="ml.g6.12xlarge",
initial_instance_count=1,
)

```

**Expected behavior**

`deploy()` should deploy the `CustomOrchestrator` as an Inference Component on the specified endpoint without error.

**Screenshots or logs**

```
│ 4226 │ │ │ # Create new IC via _deploy() │
│ ❱ 4227 │ │ │ return self._deploy( │
│ 4228 │ │ │ │ built_model=built_model, │
│ 4229 │ │ │ │ endpoint_name=endpoint_name, │
│ 4230 │ │ │ │ endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED, │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: sagemaker.serve.model_builder.ModelBuilder._deploy() got multiple values for keyword argument
'instance_type'
```

Full traceback:

```
/opt/conda/lib/python3.12/site-packages/sagemaker/serve/model_builder.py:6189 in deploy
│ ❱ 6189 │ │ │ │ │ │ self._deploy_for_ic(
│ 6190 │ │ │ │ │ │ │ ic_data=custom_orchestrator,
│ 6191 │ │ │ │ │ │ │ container_timeout_in_seconds=container_timeout_in_seconds,
│ 6192 │ │ │ │ │ │ │ instance_type=custom_orchestrator_instance_type or instance_type,

/opt/conda/lib/python3.12/site-packages/sagemaker/serve/model_builder.py:4227 in _deploy_for_ic
│ ❱ 4227 │ │ │ return self._deploy(
│ 4228 │ │ │ │ built_model=built_model,
│ 4229 │ │ │ │ endpoint_name=endpoint_name,
│ 4230 │ │ │ │ endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED,

TypeError: sagemaker.serve.model_builder.ModelBuilder._deploy() got multiple values for keyword argument 'instance_type'

```

**System information**

- **SageMaker Python SDK version**: sagemaker-serve 1.20.0 (SDK V3)
- **Framework name**: SageMaker Distribution (SMD) container
- **Framework version**: `sagemaker-distribution-prod:3.2.0-cpu`
- **Python version**: 3.12
- **CPU or GPU**: GPU (`ml.g6.2xlarge` endpoint)
- **Custom Docker image (Y/N)**: N

**Additional context**

**Root cause analysis:**

In `deploy()` ([L6189-6196](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L6189)), `_deploy_for_ic` is called with `instance_type` as an explicit kwarg:

```python
self._deploy_for_ic(
ic_data=custom_orchestrator,
container_timeout_in_seconds=container_timeout_in_seconds,
instance_type=custom_orchestrator_instance_type or instance_type, # explicit
initial_instance_count=custom_orchestrator_initial_instance_count or initial_instance_count, # explicit
endpoint_name=endpoint_name,
**kwargs,
)

```

Then in `_deploy_for_ic()` ([L4227-4237](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L4227)):

```python
def _deploy_for_ic(self, ic_data, endpoint_name, **kwargs):
...
return self._deploy(
built_model=built_model,
endpoint_name=endpoint_name,
endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED,
resources=resource_requirements,
inference_component_name=ic_name,
instance_type=kwargs.get("instance_type", self.instance_type), # extracted from kwargs
initial_instance_count=kwargs.get("initial_instance_count", 1), # extracted from kwargs
**kwargs, # ← kwargs STILL contains instance_type → duplicate!
)

```

`instance_type` is extracted from `kwargs` on one line, then `**kwargs` is spread on the next — passing the same key twice to `_deploy()`.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu tại sagemaker-serve/src/sagemaker/serve/model_builder.py, ở _deploy_for_ic() và lệnh gọi deploy() quanh các dòng 6189-6196. Tái hiện deployment CustomOrchestrator được ghi trong tài liệu, sau đó kiểm tra cách instance_type và initial_instance_count được truyền tới _deploy(). Hoàn tất khi deploy() tạo component suy luận mà không gặp TypeError do từ khóa bị trùng.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
aws, python
Lĩnh vực
cloud, machine-learning
Loại issue
Lỗi
Độ khó
2/5
Thời gian dự kiến
1-3 giờ
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
88/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.