huggingface / huggingface/smolagents

[BUG] invalid message format when using `OpenAIServerModel`

Open
#583 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
29.3k
Forks
3k
Avg merge
17m
Merged PRs (30d)
2

Description

**Describe the bug**

The latest code runs into invalid message format when using `OpenAIServerModel`

**Code to reproduce the error**
```
from smolagents import CodeAgent, DuckDuckGoSearchTool,OpenAIServerModel

model = OpenAIServerModel(
model_id="<...>",
api_base="<...>",
api_key="<...>"
)

agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)

agent.run("How many seconds would it take for a leopard at full speed to run through Pont des Arts?")
```

**Error logs (if any)**
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error in generating model output:
Error code: 400 - {'error': {'code': 'invalid_argument', 'message': 'Message index[1] content should be string', 'type': 'invalid_request_error'}}
[Step 0: Duration 0.26 seconds]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 2 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error in generating model output:
Error code: 400 - {'error': {'code': 'invalid_argument', 'message': 'Message index[1] content should be string', 'type': 'invalid_request_error'}}
```

**Expected behavior**
A clear and concise description of what you expected to happen.

**Packages version:**
`main` branch

**Additional context**

Pretty sure this is due to OpenAI SDK uses flat message format without images but nested message format with image. The current code always use nested message, which results in this bug.

without image
```
client.chat.completions.create(
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
model="gpt-4o",
)
```

with image
```
client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{
"type": "image_url",
"image_url": {"url": f"data:{img_type};base64,{img_b64_str}"},
},
],
}
],
)
```

Contributor guide

Open the contributing guide

Research direction

Start with OpenAIServerModel's message construction and compare the flat text payload with the nested image payload shown in the report. Reproduce the CodeAgent example with DuckDuckGoSearchTool, then verify that text-only requests are accepted while image messages retain their required format.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.