microsoft / microsoft/fara

Hosting: llama.cpp needs `--reasoning-format none`

Open Beginner friendly
#82 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
6.2k
Forks
603
PR merge metrics
No merged PRs in 30d

Description

Context: Filing this at the request of @swhitehead in https://huggingface.co/microsoft/Fara1.5-27B/discussions/4.

Problem: When self-hosting Fara1.5-27B (GGUF) with llama.cpp's llama-server and consuming it through any OpenAI-compatible client (I hit this via MagenticLite, but the Fara15Agent in this repo is affected the same way), the default --reasoning-format treats the model's entire output (reasoning and the <tool_call> block) as reasoning. The API response comes back with everything in reasoning_content and an empty message.content, so clients that parse the <tool_call> block out of content (e.g. src/fara/agents/fara/fara15_agent.py, which splits message.content on <tool_call>) get nothing and the agent loop breaks.

Workaround: Launch with --reasoning-format none. The full raw output (including a stray unpaired tag) is then returned in content, and tool-call parsing works normally.

Request: The README's "Hosting the Model" section covers Foundry and vLLM and notes that any OpenAI-compatible endpoint works via --base_url, but llama-server silently fails with defaults. Could you add a llama.cpp/llama-server subsection or note documenting the --reasoning-format none requirement?

Repro: llama-server -m Fara1.5-27B-Q4_K_M.gguf --port 5000 → responses have empty content, but if you add --reasoning-format none → it works. You also get a stray tag at the beginning of the model answer.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the README's “Hosting the Model” section and reproduce the llama-server commands described in the issue to observe the empty content response. Document the llama.cpp/llama-server setup, including the required reasoning-format option and the observed stray tag, so OpenAI-compatible clients can use the endpoint successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.