block / block/buzz

Local models (Ollama) with buzz-agent: undocumented, undiscoverable, needs first-class provider alias

Open
#3,145 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

Local models (Ollama, llama.cpp, vLLM, etc.) are **already usable** with `buzz-agent` through the generic OpenAI-compatible provider — but nothing in the repo says so. There is zero mention of `ollama` anywhere in the codebase or docs, and the configuration is undiscoverable. Operators reasonably assume local models are unsupported, or set env vars on the **relay** (which never calls an LLM) and conclude "it doesn't work."

## What works today (undocumented)

`buzz-agent` accepts any OpenAI-compatible endpoint:

```bash
BUZZ_AGENT_PROVIDER=openai # or "openai-compat"
OPENAI_COMPAT_BASE_URL=http://localhost:11434/v1 # Ollama
OPENAI_COMPAT_MODEL=qwen2.5:7b-instruct
OPENAI_COMPAT_API_KEY=ollama # any non-empty string; Ollama ignores it
OPENAI_COMPAT_API=chat # or auto → chat for non-*.openai.com hosts
```

Verified against a local Ollama (`qwen2.5:7b-instruct`) — chat completions work, and tool-capable models are reported via `/api/tags` (`capabilities: ["completion","tools"]`).

## The gaps

1. **Docs gap** — no `docs/` page, README section, or example covers local models. The string "ollama" appears nowhere in the repo.
2. **UX gap in `buzz-agent` config** — Ollama requires a *dummy* API key (`req("OPENAI_COMPAT_API_KEY")`), and the operator must know the magic base URL `http://localhost:11434/v1`. A first-class `BUZZ_AGENT_PROVIDER=ollama` alias could default both (key optional, base URL defaulted) while still allowing overrides.
3. **Architecture confusion** — env vars set on the `buzz-relay` container/process silently do nothing for model selection; the LLM config lives wherever the ACP runtime (`buzz-agent` et al.) runs. Worth stating explicitly in deployment docs.
4. **(Follow-up, separate PR) Desktop discoverability** — the desktop agent config offers "OpenAI-compatible" as a provider but requires an API key and gives no hint that Ollama works; an "Ollama (local)" preset with no required key and a prefilled base URL would make this one click.

## Proposal

- Add `docs/local-models.md` documenting the recipe above (Ollama first; note it applies to any OpenAI-compatible server).
- Add a first-class `ollama` provider alias in `buzz-agent` (`crates/buzz-agent/src/config.rs`): maps to the OpenAI-compatible path, defaults `OPENAI_COMPAT_BASE_URL` to `http://localhost:11434/v1`, makes the API key optional (default `ollama`), and defaults `OPENAI_COMPAT_API` to `chat`. Overrides via the existing env vars keep working.
- Unit tests for the new resolution paths.

I have a PR ready implementing exactly this.

Contributor guide

Open the contributing guide

Research direction

Start with crates/buzz-agent/src/config.rs and inspect the existing OpenAI-compatible provider resolution and its unit tests. Review the proposed docs/local-models.md scope and deployment guidance. Done means the Ollama alias has the stated defaults and overrides, the local-models documentation explains agent versus relay configuration, and unit tests cover the new resolution paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, rust
Domain
backend, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.