huggingface / huggingface/smolagents

[BUG] Prompt never properly triggers invocation of managed agent

Open
#960 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
29.3k
Forks
3k
Avg merge
17m
Merged PRs (30d)
2

Description

**Describe the bug**
I have a simple multi-agent setup as in the example (see below).

No matter what question I ask it I can never get the model to trigger the managed agent.
I am using the `o3-mini` reasoning model from Azure OpenAI.

**Code to reproduce the error**
```python
from openinference.instrumentation.smolagents import SmolagentsInstrumentor
from phoenix.otel import register
from smolagents import (
AzureOpenAIServerModel,
CodeAgent,
DuckDuckGoSearchTool,
ToolCallingAgent,
VisitWebpageTool,
)

register()
SmolagentsInstrumentor().instrument(skip_dep_check=True)

model = AzureOpenAIServerModel(
model_id="o3-mini",
api_version="2024-12-01-preview",
reasoning_effort="low",
max_completion_tokens=100000,
)

search_agent = ToolCallingAgent(
tools=[DuckDuckGoSearchTool(), VisitWebpageTool()],
model=model,
name="search_agent",
description="This is an agent that can do web search.",
)

manager_agent = ToolCallingAgent(
tools=[],
model=model,
managed_agents=[search_agent],
)
manager_agent.run(
"What is the current record of the New York Knicks NBA team? Feel free to search the web for the answer."
)
```

**Additional context**
Strangely if I copy/paste the same system prompt and user prompt into the same exact Azure OpenAI model, it behaves as expected

![Image](https://github.com/user-attachments/assets/435c93bc-9ef6-431c-83d1-6b6b3c428782)
![Image](https://github.com/user-attachments/assets/a419b3f5-b7d0-4b24-9b19-e4ea1c09434c)

Contributor guide

Open the contributing guide

Research direction

Start by running the provided reproduction with AzureOpenAIServerModel, o3-mini, and the manager_agent.run call, then inspect the ToolCallingAgent managed_agents path and the generated prompts. Compare the prompt and tool invocation behavior with the direct Azure OpenAI example. Done means the manager invokes search_agent for the Knicks question and the regression is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.