microsoft / microsoft/foundry-local

tool_choice="auto" returns tool calls as text with empty tool_calls (only "required" yields structured tool_calls)

Open
#874 6 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Summary

When calling the OpenAI-compatible `/v1/chat/completions` endpoint with `tool_choice="auto"` (the OpenAI default, and what most agent frameworks send), Foundry Local returns the tool call **as raw text inside `message.content`** with an **empty `tool_calls` array** and `finish_reason="stop"`. The structured `tool_calls` are only populated when the caller explicitly sets `tool_choice="required"`.

This breaks standard OpenAI clients and agent frameworks (e.g. anything built on the OpenAI SDK, LangChain, custom agents), because they read `message.tool_calls` and — per the OpenAI spec — expect `tool_choice="auto"` to let the model decide *and still return structured tool calls* when it chooses to call a tool. With Foundry Local they instead receive prose containing an XML/JSON blob that they cannot dispatch.

### Reproduction

Model: `qwen2.5-coder-7b-instruct-generic-cpu` (also reproduces on other tool-capable models), Foundry Local `0.10.0`.

Request (only `tool_choice` differs):

```bash
curl -s http://127.0.0.1:/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "qwen2.5-coder-7b-instruct-generic-cpu",
"messages": [{"role": "user", "content": "What is the weather in Paris? Use the tool."}],
"tools": [{"type": "function", "function": {
"name": "get_weather",
"description": "Get weather for a city",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
"tool_choice": "auto",
"temperature": 0
}'
```

### Actual behavior

**`tool_choice="auto"`** — tool call leaks into content as text, `tool_calls` empty:

```
finish_reason: stop
tool_calls: []
content: "```xml\n\n {\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\"}}\n\n```"
```

**`tool_choice="required"`** — structured tool call is returned as expected:

```
finish_reason: tool_calls
tool_calls: [{"index": 0, "id": "09xikmhSQ", "type": "function",
"function": {"name": "get_weather", "arguments": "{\n \"city\": \"Paris\"\n}"}}]
```

### Expected behavior

With `tool_choice="auto"`, when the model decides to call a tool, Foundry Local should parse the model's tool-call output and populate `message.tool_calls` with `finish_reason="tool_calls"` — the same as `tool_choice="required"` — instead of returning the call as raw text. `auto` should differ from `required` only in *whether* a tool must be called, not in *how* a chosen call is surfaced.

### Impact

- Any OpenAI-compatible agent/framework that sends the default `tool_choice="auto"` cannot use Foundry Local tool calling without a custom text-scraping workaround.
- Forcing `tool_choice="required"` is not a general fix, because it prevents the model from answering directly when no tool is needed.

### Environment

- Foundry Local: `0.10.0`
- Endpoint: OpenAI-compatible `/v1/chat/completions`
- Models: `qwen2.5-coder-7b-instruct-generic-cpu` (reproduces broadly across tool-capable models)
- OS: Linux

### Question

Is the `auto` → text vs `required` → structured behavior intentional, or a gap in the server-side tool-call parsing? Ideally `auto` would run the same tool-call extraction that `required` already does and emit structured `tool_calls`.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the documented /v1/chat/completions curl request, comparing tool_choice="auto" with "required". Trace the server-side tool-call parsing and response construction; done means auto produces structured message.tool_calls and finish_reason="tool_calls" when the model chooses a tool, while still allowing a direct response when no tool is needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, api, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.