microsoft / microsoft/magentic-ui

[bug] OmniAgent drops MagenticBrain tool calls when the server peels them into message.tool_calls

Open
#573 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
10.1k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

OmniAgent only parses MagenticBrain tool use from `message.content` (`{...}` XML). Some OpenAI-compatible servers detect those markers, move them into native `message.tool_calls`, and leave only the model's prose in `content`.

OmniAgent then treats the turn as bare text: one nudge, then a "final answer" that narrates using the tool instead of calling it. Sub-agents (e.g. Fara via `delegate_cua`) never start.

This is not MagenticBrain failing to emit tool calls — the server already has them in `tool_calls`. The client just never reads that field.

The documented vLLM / Hugging Face / Foundry path is likely unaffected (those servers typically leave the XML in `content`). The bug shows up with servers that implement native tool-call extraction, including `mlx_lm.server`.

### Steps to reproduce

1. Serve MagenticBrain through an OpenAI-compatible endpoint that extracts `` into `message.tool_calls` (reproduced with `mlx_lm.server --model mlx-community/MagenticBrain-8bit`).
2. Point MagenticLite's orchestrator `base_url` at that `/v1` endpoint.
3. Ask a task that should dispatch the web agent, e.g. "Go to https://example.com and describe the homepage."
4. Optionally confirm the server *did* emit native tool calls:

```python
from openai import OpenAI
c = OpenAI(base_url="http://127.0.0.1:8100/v1", api_key="not-needed")
r = c.chat.completions.create(
model="mlx-community/MagenticBrain-8bit",
messages=[
{"role": "system", "content": "...MagenticLite tool prompt with delegate_cua..."},
{"role": "user", "content": "Go to https://example.com and describe the homepage."},
],
)
print(r.choices[0].message.content)
print(r.choices[0].message.tool_calls)
print(r.choices[0].finish_reason) # often "tool_calls"

### Expected behavior

OmniAgent parses a delegate_cua (or other) tool call and runs it. The UI shows Using tool: delegate_cua, and Fara starts.

### Actual behavior

### **UI loop:**

• **Reasoning**: "This is a web browsing task, so I should use the delegate_cua function..."
• **Nudge**: response did not contain or
• **Final answer**: "I need to delegate the web browsing task to the web agent. Let me use the delegate_cua function with the appropriate parameters."

Fara never runs. Trace shows content is the prose only; finish_reason on the raw completion is tool_calls.

### MagenticLite version

0.2.2

### Operating system

macOS

### Python version

3.12

### Model client / provider

`mlx-community/MagenticBrain-8bit` via `mlx_lm.server` & `mlx-community/Fara1.5-9B-8bit` via `mlx_vlm.server`

### Additional context

**Root cause**: `OmniResponses._call_api` takes `response.choices[0].message.content or ""` and `parse_response()` only looks for `` in that string. Native `message.tool_calls` is ignored.

**Suggested fix**: If `tool_calls` is present and `content` does not already contain `` tags, reconstruct the XML MagenticLite already understands, then parse as today. If content already has the tags (vLLM path), leave it alone.

I prototyped this in `zboyles/magentic-ui` branch `fix/rehydrate-native-tool-calls` (commit [ece97a2](https://github.com/zboyles/magentic-ui/commit/ece97a2)): `rehydrate_native_tool_calls()` in `_parse.py`, used from `_responses.py`, plus tests in `tests/agents/omni/test_parse.py`. Ran several live checks successfully, the same browsing task emitted `delegate_cua` and started Fara.

I can adapt that patch if you want to take it in-tree; I created this issue first per `CONTRIBUTING.md`.

Contributor guide

Open the contributing guide

Research direction

Start with OmniResponses._call_api in _responses.py and the response parsing in _parse.py; compare how message.content and native message.tool_calls are handled. Read the existing tests in tests/agents/omni/test_parse.py and run them first. Done means native tool calls are parsed without breaking content that already contains XML tool-call tags, with tests covering both paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend-api-design
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.