coder / coder/xum

LiteLLM + Bedrock issue

Open
#3,182 0 comments 0 reactions 0 assignees View on GitHub
third-party
Dominant language
TypeScript
Stars
2k
Forks
134
Avg merge
14h 57m
Merged PRs (30d)
307

Description

I'm trying to setup MUX to work with our LiteLLM server. LiteLLM is exposes the claude sonnet 4.6 bedrock endpoint.

I have set it up the provider:
```
{
"openai": {
"apiKey": "sk-U...w",
"baseUrl": "https://litellm.our.domain/v1",
"models": [
"claude-sonnet"
]
}
}
```

I've tried the following model configs for LiteLLM

```
- model_name: claude-sonnet
litellm_params:
model: bedrock/anthropic.claude-sonnet-4-6
model_id: arn:aws:bedrock:us-east-1:6....8:application-inference-profile/75irbw5rdcvw
aws_region_name: os.environ/AWS_REGION_NAME
aws_role_name: os.environ/AWS_ROLE_ARN
aws_session_name: litellm-bedrock
```
and
```
- model_name: claude-sonnet-4-6-mux
litellm_params:
model: bedrock/anthropic.claude-sonnet-4-6
model_id: arn:aws:bedrock:us-east-1:^...8:application-inference-profile/75irbw5rdcvw
aws_region_name: os.environ/AWS_REGION_NAME
aws_role_name: os.environ/AWS_ROLE_ARN
aws_session_name: litellm-bedrock
drop_params: true
include_reasoning: false
```

I've tried enabling and disaling:
```
drop_params: true
include_reasoning: false
```

Sometimes I get errors like:
```
text part chatcmpl-b7e252dd-c0ff-4c84-8f63-a7d63e477719 not found
```

Sometimes it works - but mid way through a response I get errors like:
```
Stream Error
api

litellm.BadRequestError: BedrockException - {"message":"The number of toolResult blocks at messages.4.content exceeds the number of toolUse blocks of previous turn."}. Received Model Group=claude-sonnet
Available Model Group Fallbacks=['bedrock-claude-sonnet-4-5']
Error doing the fallback: litellm.BadRequestError: BedrockException - {"message":"The number of toolResult blocks at messages.4.content exceeds the number of toolUse blocks of previous turn."}No fallback model group found for original model_group=bedrock-claude-sonnet-4-5. Fallbacks=[{'claude-opus': ['bedrock-claude-opus-4-5']}, {'claude-sonnet': ['bedrock-claude-sonnet-4-5']}, {'claude-haiku': ['bedrock-claude-haiku-4-5']}]. Received Model Group=bedrock-claude-sonnet-4-5
Available Model Group Fallbacks=None
Error doing the fallback: litellm.BadRequestError: BedrockException - {"message":"The number of toolResult blocks at messages.4.content exceeds the number of toolUse blocks of previous turn."}No fallback model group found for original model_group=bedrock-claude-sonnet-4-5. Fallbacks=[{'claude-opus': ['bedrock-claude-opus-4-5']}, {'claude-sonnet': ['bedrock-claude-sonnet-4-5']}, {'claude-haiku': ['bedrock-claude-haiku-4-5']}]
```

Is there anything I need to do to get this working correctly?

Here's a screenshot:

Image

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.