aws / aws/sagemaker-python-sdk
`_deploy_for_ic` passes `instance_type` twice to `_deploy()` causing `TypeError` when deploying `CustomOrchestrator` as Inference Component
- Ngôn ngữ chính
- Python
- Star
- 2.3k
- Fork
- 1.3k
- Merge trung bình
- 1 ngày 22 giờ
- Pull request đã merge (30 ngày)
- 35
Mô tả
**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
**Describe the bug**
When deploying a `CustomOrchestrator` as an Inference Component — as documented in the [Build and deploy AI inference workflows with new enhancements to the Amazon SageMaker Python SDK](https://aws.amazon.com/blogs/machine-learning/build-and-deploy-ai-inference-workflows-with-new-enhancements-to-the-amazon-sagemaker-python-sdk/) blog post and the [Llama3.1-Mistral reference notebook](https://github.com/aws-samples/sagemaker-genai-hosting-examples/blob/main/Llama3.1-Mistral-workflow/Llama3.1-8B-Mistral-7B-inference-orchestrator.ipynb) — calling `deploy()` raises a `TypeError: got multiple values for keyword argument 'instance_type'`.
The issue is in `_deploy_for_ic()` ([model_builder.py L4227-4237](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L4227)): `instance_type` and `initial_instance_count` are passed **both** as explicit keyword arguments **and** via `**kwargs` spread to `self._deploy()`.
**To reproduce**
```python
from sagemaker.serve.model_builder import ModelBuilder, SchemaBuilder
from sagemaker.serve.spec.inference_base import CustomOrchestrator
from sagemaker.core.inference_config import ResourceRequirements
from sagemaker.core.helper.session_helper import Session, get_execution_role
class MyOrchestrator(CustomOrchestrator):
def __init__(self, endpoint_name, component_names):
super().__init__()
self.endpoint_name = endpoint_name
self.component_names = component_names
def handle(self, data, context=None):
import json
response = self.client.invoke_endpoint(
EndpointName=self.endpoint_name,
InferenceComponentName=self.component_names[0],
Body=data if isinstance(data, (str, bytes)) else json.dumps(data),
ContentType="application/json"
)
return json.loads(response["Body"].read())
role = get_execution_role()
sess = Session()
# Step 1: Build the orchestrator
orchestrator = ModelBuilder(
inference_spec=MyOrchestrator(
endpoint_name="my-existing-endpoint",
component_names=["base-ic", "adapter-ic"],
),
dependencies={"auto": False, "custom": ["cloudpickle"]},
sagemaker_session=sess,
role_arn=role,
schema_builder=SchemaBuilder(sample_input="Test", sample_output={"generated_text": "test"}),
)
# Workaround for missing constructor fields (separate issue)
orchestrator.resource_requirements = ResourceRequirements(
requests={"memory": 4096, "num_accelerators": 1, "copies": 1, "num_cpus": 2}
)
orchestrator.inference_component_name = "my-orchestrator-ic"
orchestrator.build()
# Step 2: Deploy — this triggers the bug
orchestrator.deploy(
endpoint_name="my-existing-endpoint",
custom_orchestrator_instance_type="ml.g6.12xlarge",
initial_instance_count=1,
)
```
**Expected behavior**
`deploy()` should deploy the `CustomOrchestrator` as an Inference Component on the specified endpoint without error.
**Screenshots or logs**
```
│ 4226 │ │ │ # Create new IC via _deploy() │
│ ❱ 4227 │ │ │ return self._deploy( │
│ 4228 │ │ │ │ built_model=built_model, │
│ 4229 │ │ │ │ endpoint_name=endpoint_name, │
│ 4230 │ │ │ │ endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED, │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: sagemaker.serve.model_builder.ModelBuilder._deploy() got multiple values for keyword argument
'instance_type'
```
Full traceback:
```
/opt/conda/lib/python3.12/site-packages/sagemaker/serve/model_builder.py:6189 in deploy
│ ❱ 6189 │ │ │ │ │ │ self._deploy_for_ic(
│ 6190 │ │ │ │ │ │ │ ic_data=custom_orchestrator,
│ 6191 │ │ │ │ │ │ │ container_timeout_in_seconds=container_timeout_in_seconds,
│ 6192 │ │ │ │ │ │ │ instance_type=custom_orchestrator_instance_type or instance_type,
/opt/conda/lib/python3.12/site-packages/sagemaker/serve/model_builder.py:4227 in _deploy_for_ic
│ ❱ 4227 │ │ │ return self._deploy(
│ 4228 │ │ │ │ built_model=built_model,
│ 4229 │ │ │ │ endpoint_name=endpoint_name,
│ 4230 │ │ │ │ endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED,
TypeError: sagemaker.serve.model_builder.ModelBuilder._deploy() got multiple values for keyword argument 'instance_type'
```
**System information**
- **SageMaker Python SDK version**: sagemaker-serve 1.20.0 (SDK V3)
- **Framework name**: SageMaker Distribution (SMD) container
- **Framework version**: `sagemaker-distribution-prod:3.2.0-cpu`
- **Python version**: 3.12
- **CPU or GPU**: GPU (`ml.g6.2xlarge` endpoint)
- **Custom Docker image (Y/N)**: N
**Additional context**
**Root cause analysis:**
In `deploy()` ([L6189-6196](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L6189)), `_deploy_for_ic` is called with `instance_type` as an explicit kwarg:
```python
self._deploy_for_ic(
ic_data=custom_orchestrator,
container_timeout_in_seconds=container_timeout_in_seconds,
instance_type=custom_orchestrator_instance_type or instance_type, # explicit
initial_instance_count=custom_orchestrator_initial_instance_count or initial_instance_count, # explicit
endpoint_name=endpoint_name,
**kwargs,
)
```
Then in `_deploy_for_ic()` ([L4227-4237](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L4227)):
```python
def _deploy_for_ic(self, ic_data, endpoint_name, **kwargs):
...
return self._deploy(
built_model=built_model,
endpoint_name=endpoint_name,
endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED,
resources=resource_requirements,
inference_component_name=ic_name,
instance_type=kwargs.get("instance_type", self.instance_type), # extracted from kwargs
initial_instance_count=kwargs.get("initial_instance_count", 1), # extracted from kwargs
**kwargs, # ← kwargs STILL contains instance_type → duplicate!
)
```
`instance_type` is extracted from `kwargs` on one line, then `**kwargs` is spread on the next — passing the same key twice to `_deploy()`.
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu tại sagemaker-serve/src/sagemaker/serve/model_builder.py, ở _deploy_for_ic() và lệnh gọi deploy() quanh các dòng 6189-6196. Tái hiện deployment CustomOrchestrator được ghi trong tài liệu, sau đó kiểm tra cách instance_type và initial_instance_count được truyền tới _deploy(). Hoàn tất khi deploy() tạo component suy luận mà không gặp TypeError do từ khóa bị trùng.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- aws, python
- Lĩnh vực
- cloud, machine-learning
- Loại issue
- Lỗi
- Độ khó
- 2/5
- Thời gian dự kiến
- 1-3 giờ
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 88/100