Local models (Ollama) with buzz-agent: undocumented, undiscoverable, needs first-class provider alias
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
Local models (Ollama, llama.cpp, vLLM, etc.) are **already usable** with `buzz-agent` through the generic OpenAI-compatible provider — but nothing in the repo says so. There is zero mention of `ollama` anywhere in the codebase or docs, and the configuration is undiscoverable. Operators reasonably assume local models are unsupported, or set env vars on the **relay** (which never calls an LLM) and conclude "it doesn't work."
## What works today (undocumented)
`buzz-agent` accepts any OpenAI-compatible endpoint:
```bash
BUZZ_AGENT_PROVIDER=openai # or "openai-compat"
OPENAI_COMPAT_BASE_URL=http://localhost:11434/v1 # Ollama
OPENAI_COMPAT_MODEL=qwen2.5:7b-instruct
OPENAI_COMPAT_API_KEY=ollama # any non-empty string; Ollama ignores it
OPENAI_COMPAT_API=chat # or auto → chat for non-*.openai.com hosts
```
Verified against a local Ollama (`qwen2.5:7b-instruct`) — chat completions work, and tool-capable models are reported via `/api/tags` (`capabilities: ["completion","tools"]`).
## The gaps
1. **Docs gap** — no `docs/` page, README section, or example covers local models. The string "ollama" appears nowhere in the repo.
2. **UX gap in `buzz-agent` config** — Ollama requires a *dummy* API key (`req("OPENAI_COMPAT_API_KEY")`), and the operator must know the magic base URL `http://localhost:11434/v1`. A first-class `BUZZ_AGENT_PROVIDER=ollama` alias could default both (key optional, base URL defaulted) while still allowing overrides.
3. **Architecture confusion** — env vars set on the `buzz-relay` container/process silently do nothing for model selection; the LLM config lives wherever the ACP runtime (`buzz-agent` et al.) runs. Worth stating explicitly in deployment docs.
4. **(Follow-up, separate PR) Desktop discoverability** — the desktop agent config offers "OpenAI-compatible" as a provider but requires an API key and gives no hint that Ollama works; an "Ollama (local)" preset with no required key and a prefilled base URL would make this one click.
## Proposal
- Add `docs/local-models.md` documenting the recipe above (Ollama first; note it applies to any OpenAI-compatible server).
- Add a first-class `ollama` provider alias in `buzz-agent` (`crates/buzz-agent/src/config.rs`): maps to the OpenAI-compatible path, defaults `OPENAI_COMPAT_BASE_URL` to `http://localhost:11434/v1`, makes the API key optional (default `ollama`), and defaults `OPENAI_COMPAT_API` to `chat`. Overrides via the existing env vars keep working.
- Unit tests for the new resolution paths.
I have a PR ready implementing exactly this.
Contributor guide
Research direction
Start with crates/buzz-agent/src/config.rs and inspect the existing OpenAI-compatible provider resolution and its unit tests. Review the proposed docs/local-models.md scope and deployment guidance. Done means the Ollama alias has the stated defaults and overrides, the local-models documentation explains agent versus relay configuration, and unit tests cover the new resolution paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ollama, rust
- Domain
- backend, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100