aws / aws/sagemaker-python-sdk
`_deploy_for_ic` passes `instance_type` twice to `_deploy()` causing `TypeError` when deploying `CustomOrchestrator` as Inference Component
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 35
Description
**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
**Describe the bug**
When deploying a `CustomOrchestrator` as an Inference Component — as documented in the [Build and deploy AI inference workflows with new enhancements to the Amazon SageMaker Python SDK](https://aws.amazon.com/blogs/machine-learning/build-and-deploy-ai-inference-workflows-with-new-enhancements-to-the-amazon-sagemaker-python-sdk/) blog post and the [Llama3.1-Mistral reference notebook](https://github.com/aws-samples/sagemaker-genai-hosting-examples/blob/main/Llama3.1-Mistral-workflow/Llama3.1-8B-Mistral-7B-inference-orchestrator.ipynb) — calling `deploy()` raises a `TypeError: got multiple values for keyword argument 'instance_type'`.
The issue is in `_deploy_for_ic()` ([model_builder.py L4227-4237](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L4227)): `instance_type` and `initial_instance_count` are passed **both** as explicit keyword arguments **and** via `**kwargs` spread to `self._deploy()`.
**To reproduce**
```python
from sagemaker.serve.model_builder import ModelBuilder, SchemaBuilder
from sagemaker.serve.spec.inference_base import CustomOrchestrator
from sagemaker.core.inference_config import ResourceRequirements
from sagemaker.core.helper.session_helper import Session, get_execution_role
class MyOrchestrator(CustomOrchestrator):
def __init__(self, endpoint_name, component_names):
super().__init__()
self.endpoint_name = endpoint_name
self.component_names = component_names
def handle(self, data, context=None):
import json
response = self.client.invoke_endpoint(
EndpointName=self.endpoint_name,
InferenceComponentName=self.component_names[0],
Body=data if isinstance(data, (str, bytes)) else json.dumps(data),
ContentType="application/json"
)
return json.loads(response["Body"].read())
role = get_execution_role()
sess = Session()
# Step 1: Build the orchestrator
orchestrator = ModelBuilder(
inference_spec=MyOrchestrator(
endpoint_name="my-existing-endpoint",
component_names=["base-ic", "adapter-ic"],
),
dependencies={"auto": False, "custom": ["cloudpickle"]},
sagemaker_session=sess,
role_arn=role,
schema_builder=SchemaBuilder(sample_input="Test", sample_output={"generated_text": "test"}),
)
# Workaround for missing constructor fields (separate issue)
orchestrator.resource_requirements = ResourceRequirements(
requests={"memory": 4096, "num_accelerators": 1, "copies": 1, "num_cpus": 2}
)
orchestrator.inference_component_name = "my-orchestrator-ic"
orchestrator.build()
# Step 2: Deploy — this triggers the bug
orchestrator.deploy(
endpoint_name="my-existing-endpoint",
custom_orchestrator_instance_type="ml.g6.12xlarge",
initial_instance_count=1,
)
```
**Expected behavior**
`deploy()` should deploy the `CustomOrchestrator` as an Inference Component on the specified endpoint without error.
**Screenshots or logs**
```
│ 4226 │ │ │ # Create new IC via _deploy() │
│ ❱ 4227 │ │ │ return self._deploy( │
│ 4228 │ │ │ │ built_model=built_model, │
│ 4229 │ │ │ │ endpoint_name=endpoint_name, │
│ 4230 │ │ │ │ endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED, │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: sagemaker.serve.model_builder.ModelBuilder._deploy() got multiple values for keyword argument
'instance_type'
```
Full traceback:
```
/opt/conda/lib/python3.12/site-packages/sagemaker/serve/model_builder.py:6189 in deploy
│ ❱ 6189 │ │ │ │ │ │ self._deploy_for_ic(
│ 6190 │ │ │ │ │ │ │ ic_data=custom_orchestrator,
│ 6191 │ │ │ │ │ │ │ container_timeout_in_seconds=container_timeout_in_seconds,
│ 6192 │ │ │ │ │ │ │ instance_type=custom_orchestrator_instance_type or instance_type,
/opt/conda/lib/python3.12/site-packages/sagemaker/serve/model_builder.py:4227 in _deploy_for_ic
│ ❱ 4227 │ │ │ return self._deploy(
│ 4228 │ │ │ │ built_model=built_model,
│ 4229 │ │ │ │ endpoint_name=endpoint_name,
│ 4230 │ │ │ │ endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED,
TypeError: sagemaker.serve.model_builder.ModelBuilder._deploy() got multiple values for keyword argument 'instance_type'
```
**System information**
- **SageMaker Python SDK version**: sagemaker-serve 1.20.0 (SDK V3)
- **Framework name**: SageMaker Distribution (SMD) container
- **Framework version**: `sagemaker-distribution-prod:3.2.0-cpu`
- **Python version**: 3.12
- **CPU or GPU**: GPU (`ml.g6.2xlarge` endpoint)
- **Custom Docker image (Y/N)**: N
**Additional context**
**Root cause analysis:**
In `deploy()` ([L6189-6196](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L6189)), `_deploy_for_ic` is called with `instance_type` as an explicit kwarg:
```python
self._deploy_for_ic(
ic_data=custom_orchestrator,
container_timeout_in_seconds=container_timeout_in_seconds,
instance_type=custom_orchestrator_instance_type or instance_type, # explicit
initial_instance_count=custom_orchestrator_initial_instance_count or initial_instance_count, # explicit
endpoint_name=endpoint_name,
**kwargs,
)
```
Then in `_deploy_for_ic()` ([L4227-4237](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L4227)):
```python
def _deploy_for_ic(self, ic_data, endpoint_name, **kwargs):
...
return self._deploy(
built_model=built_model,
endpoint_name=endpoint_name,
endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED,
resources=resource_requirements,
inference_component_name=ic_name,
instance_type=kwargs.get("instance_type", self.instance_type), # extracted from kwargs
initial_instance_count=kwargs.get("initial_instance_count", 1), # extracted from kwargs
**kwargs, # ← kwargs STILL contains instance_type → duplicate!
)
```
`instance_type` is extracted from `kwargs` on one line, then `**kwargs` is spread on the next — passing the same key twice to `_deploy()`.
Contributor guide
Research direction
Start in sagemaker-serve/src/sagemaker/serve/model_builder.py at _deploy_for_ic() and the deploy() call around lines 6189-6196. Reproduce the documented CustomOrchestrator deployment, then inspect how instance_type and initial_instance_count reach _deploy(). Done means deploy() creates the inference component without the duplicate-keyword TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100