Hosting: llama.cpp needs `--reasoning-format none`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 603
- PR merge metrics
- No merged PRs in 30d
Description
Context: Filing this at the request of @swhitehead in https://huggingface.co/microsoft/Fara1.5-27B/discussions/4.
Problem: When self-hosting Fara1.5-27B (GGUF) with llama.cpp's llama-server and consuming it through any OpenAI-compatible client (I hit this via MagenticLite, but the Fara15Agent in this repo is affected the same way), the default --reasoning-format treats the model's entire output (reasoning and the <tool_call> block) as reasoning. The API response comes back with everything in reasoning_content and an empty message.content, so clients that parse the <tool_call> block out of content (e.g. src/fara/agents/fara/fara15_agent.py, which splits message.content on <tool_call>) get nothing and the agent loop breaks.
Workaround: Launch with --reasoning-format none. The full raw output (including a stray unpaired tag) is then returned in content, and tool-call parsing works normally.
Request: The README's "Hosting the Model" section covers Foundry and vLLM and notes that any OpenAI-compatible endpoint works via --base_url, but llama-server silently fails with defaults. Could you add a llama.cpp/llama-server subsection or note documenting the --reasoning-format none requirement?
Repro: llama-server -m Fara1.5-27B-Q4_K_M.gguf --port 5000 → responses have empty content, but if you add --reasoning-format none → it works. You also get a stray tag at the beginning of the model answer.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the README's “Hosting the Model” section and reproduce the llama-server commands described in the issue to observe the empty content response. Document the llama.cpp/llama-server setup, including the required reasoning-format option and the observed stray tag, so OpenAI-compatible clients can use the endpoint successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100