huggingface / huggingface/smolagents
[BUG] invalid message format when using `OpenAIServerModel`
- Dominant language
- Python
- Stars
- 29.3k
- Forks
- 3k
- Avg merge
- 17m
- Merged PRs (30d)
- 2
Description
**Describe the bug**
The latest code runs into invalid message format when using `OpenAIServerModel`
**Code to reproduce the error**
```
from smolagents import CodeAgent, DuckDuckGoSearchTool,OpenAIServerModel
model = OpenAIServerModel(
model_id="<...>",
api_base="<...>",
api_key="<...>"
)
agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)
agent.run("How many seconds would it take for a leopard at full speed to run through Pont des Arts?")
```
**Error logs (if any)**
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error in generating model output:
Error code: 400 - {'error': {'code': 'invalid_argument', 'message': 'Message index[1] content should be string', 'type': 'invalid_request_error'}}
[Step 0: Duration 0.26 seconds]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 2 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error in generating model output:
Error code: 400 - {'error': {'code': 'invalid_argument', 'message': 'Message index[1] content should be string', 'type': 'invalid_request_error'}}
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Packages version:**
`main` branch
**Additional context**
Pretty sure this is due to OpenAI SDK uses flat message format without images but nested message format with image. The current code always use nested message, which results in this bug.
without image
```
client.chat.completions.create(
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
model="gpt-4o",
)
```
with image
```
client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{
"type": "image_url",
"image_url": {"url": f"data:{img_type};base64,{img_b64_str}"},
},
],
}
],
)
```
Contributor guide
Research direction
Start with OpenAIServerModel's message construction and compare the flat text payload with the nested image payload shown in the report. Reproduce the CodeAgent example with DuckDuckGoSearchTool, then verify that text-only requests are accepted while image messages retain their required format.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100