aws / aws/sagemaker-python-sdk

`_deploy_for_ic` passes `instance_type` twice to `_deploy()` causing `TypeError` when deploying `CustomOrchestrator` as Inference Component

Open Beginner friendly
#6,199 0 comments 0 reactions 0 assignees View on GitHub
component: model builder type: bug
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

**Describe the bug**

When deploying a `CustomOrchestrator` as an Inference Component — as documented in the [Build and deploy AI inference workflows with new enhancements to the Amazon SageMaker Python SDK](https://aws.amazon.com/blogs/machine-learning/build-and-deploy-ai-inference-workflows-with-new-enhancements-to-the-amazon-sagemaker-python-sdk/) blog post and the [Llama3.1-Mistral reference notebook](https://github.com/aws-samples/sagemaker-genai-hosting-examples/blob/main/Llama3.1-Mistral-workflow/Llama3.1-8B-Mistral-7B-inference-orchestrator.ipynb) — calling `deploy()` raises a `TypeError: got multiple values for keyword argument 'instance_type'`.

The issue is in `_deploy_for_ic()` ([model_builder.py L4227-4237](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L4227)): `instance_type` and `initial_instance_count` are passed **both** as explicit keyword arguments **and** via `**kwargs` spread to `self._deploy()`.

**To reproduce**

```python
from sagemaker.serve.model_builder import ModelBuilder, SchemaBuilder
from sagemaker.serve.spec.inference_base import CustomOrchestrator
from sagemaker.core.inference_config import ResourceRequirements
from sagemaker.core.helper.session_helper import Session, get_execution_role

class MyOrchestrator(CustomOrchestrator):
def __init__(self, endpoint_name, component_names):
super().__init__()
self.endpoint_name = endpoint_name
self.component_names = component_names

def handle(self, data, context=None):
import json
response = self.client.invoke_endpoint(
EndpointName=self.endpoint_name,
InferenceComponentName=self.component_names[0],
Body=data if isinstance(data, (str, bytes)) else json.dumps(data),
ContentType="application/json"
)
return json.loads(response["Body"].read())

role = get_execution_role()
sess = Session()

# Step 1: Build the orchestrator
orchestrator = ModelBuilder(
inference_spec=MyOrchestrator(
endpoint_name="my-existing-endpoint",
component_names=["base-ic", "adapter-ic"],
),
dependencies={"auto": False, "custom": ["cloudpickle"]},
sagemaker_session=sess,
role_arn=role,
schema_builder=SchemaBuilder(sample_input="Test", sample_output={"generated_text": "test"}),
)

# Workaround for missing constructor fields (separate issue)
orchestrator.resource_requirements = ResourceRequirements(
requests={"memory": 4096, "num_accelerators": 1, "copies": 1, "num_cpus": 2}
)
orchestrator.inference_component_name = "my-orchestrator-ic"

orchestrator.build()

# Step 2: Deploy — this triggers the bug
orchestrator.deploy(
endpoint_name="my-existing-endpoint",
custom_orchestrator_instance_type="ml.g6.12xlarge",
initial_instance_count=1,
)

```

**Expected behavior**

`deploy()` should deploy the `CustomOrchestrator` as an Inference Component on the specified endpoint without error.

**Screenshots or logs**

```
│ 4226 │ │ │ # Create new IC via _deploy() │
│ ❱ 4227 │ │ │ return self._deploy( │
│ 4228 │ │ │ │ built_model=built_model, │
│ 4229 │ │ │ │ endpoint_name=endpoint_name, │
│ 4230 │ │ │ │ endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED, │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: sagemaker.serve.model_builder.ModelBuilder._deploy() got multiple values for keyword argument
'instance_type'
```

Full traceback:

```
/opt/conda/lib/python3.12/site-packages/sagemaker/serve/model_builder.py:6189 in deploy
│ ❱ 6189 │ │ │ │ │ │ self._deploy_for_ic(
│ 6190 │ │ │ │ │ │ │ ic_data=custom_orchestrator,
│ 6191 │ │ │ │ │ │ │ container_timeout_in_seconds=container_timeout_in_seconds,
│ 6192 │ │ │ │ │ │ │ instance_type=custom_orchestrator_instance_type or instance_type,

/opt/conda/lib/python3.12/site-packages/sagemaker/serve/model_builder.py:4227 in _deploy_for_ic
│ ❱ 4227 │ │ │ return self._deploy(
│ 4228 │ │ │ │ built_model=built_model,
│ 4229 │ │ │ │ endpoint_name=endpoint_name,
│ 4230 │ │ │ │ endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED,

TypeError: sagemaker.serve.model_builder.ModelBuilder._deploy() got multiple values for keyword argument 'instance_type'

```

**System information**

- **SageMaker Python SDK version**: sagemaker-serve 1.20.0 (SDK V3)
- **Framework name**: SageMaker Distribution (SMD) container
- **Framework version**: `sagemaker-distribution-prod:3.2.0-cpu`
- **Python version**: 3.12
- **CPU or GPU**: GPU (`ml.g6.2xlarge` endpoint)
- **Custom Docker image (Y/N)**: N

**Additional context**

**Root cause analysis:**

In `deploy()` ([L6189-6196](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L6189)), `_deploy_for_ic` is called with `instance_type` as an explicit kwarg:

```python
self._deploy_for_ic(
ic_data=custom_orchestrator,
container_timeout_in_seconds=container_timeout_in_seconds,
instance_type=custom_orchestrator_instance_type or instance_type, # explicit
initial_instance_count=custom_orchestrator_initial_instance_count or initial_instance_count, # explicit
endpoint_name=endpoint_name,
**kwargs,
)

```

Then in `_deploy_for_ic()` ([L4227-4237](https://github.com/aws/sagemaker-python-sdk/blob/44410e2c55202f80d980e1dfd4533e66b9d44495/sagemaker-serve/src/sagemaker/serve/model_builder.py#L4227)):

```python
def _deploy_for_ic(self, ic_data, endpoint_name, **kwargs):
...
return self._deploy(
built_model=built_model,
endpoint_name=endpoint_name,
endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED,
resources=resource_requirements,
inference_component_name=ic_name,
instance_type=kwargs.get("instance_type", self.instance_type), # extracted from kwargs
initial_instance_count=kwargs.get("initial_instance_count", 1), # extracted from kwargs
**kwargs, # ← kwargs STILL contains instance_type → duplicate!
)

```

`instance_type` is extracted from `kwargs` on one line, then `**kwargs` is spread on the next — passing the same key twice to `_deploy()`.

Contributor guide

Open the contributing guide

Research direction

Start in sagemaker-serve/src/sagemaker/serve/model_builder.py at _deploy_for_ic() and the deploy() call around lines 6189-6196. Reproduce the documented CustomOrchestrator deployment, then inspect how instance_type and initial_instance_count reach _deploy(). Done means deploy() creates the inference component without the duplicate-keyword TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.