bench: qualify bleeding-edge local inference backends on real agent work
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Problem
Buzz can route agents to arbitrary OpenAI-compatible endpoints, and the Harbor Buzz Orchestra benchmark can swap endpoint deployment configuration. It does not yet produce a backend qualification receipt that makes rapidly changing local inference engines comparable and upstream-debuggable.
On Apple Silicon, backend/model changes can materially improve throughput while also regressing batching isolation, streaming tool calls, reasoning controls, cancellation, prefix caches, or long-running server health. Tokens per second alone is not a sufficient promotion gate for an agent workforce.
## Proposed first slice
Add a benchmark-only qualification path that:
- records exact backend, model, drafter, dependency, OS, hardware, and launch configuration provenance;
- exercises an OpenAI-compatible endpoint with focused correctness/concurrency probes before agent work;
- emits a machine-readable, append-only result receipt with raw evidence paths;
- keeps endpoint/backend selection outside Buzz product semantics;
- establishes a Qwen3.8-27B autoregressive MLX baseline before comparing MTP and other serving engines;
- can feed surviving conditions into the existing Harbor Buzz Orchestra workload harness.
## Initial acceptance criteria
- [ ] A local endpoint condition can be described without hard-coding MLX concepts into Buzz runtime code.
- [ ] The qualifier checks basic generation, streaming, reasoning controls, tool-call continuation, cancellation, concurrency isolation, and liveness after failures where supported.
- [ ] Unsupported capabilities are recorded explicitly rather than treated as passes.
- [ ] Every run emits a JSON receipt pinning revisions and configuration.
- [ ] Unit tests cover manifest validation and receipt serialization.
- [ ] The first checked-in example documents the source-pinned `mlx-vlm` autoregressive baseline procedure for `Qwen3.8-27B-8bit`.
## Non-goals
- Selecting a permanent inference engine.
- Adding MLX-specific concepts to the Buzz agent protocol.
- Publishing benchmark claims before repeated real-workload validation.
- Modifying or redistributing model weights.
Contributor guide
Research direction
Start by locating the existing Harbor Buzz Orchestra workload harness and the benchmark entry point that can describe endpoint conditions. Read how benchmark configuration and results are currently represented, then trace the requested manifest validation and receipt serialization tests. Done means a benchmark-only qualifier records provenance, probe outcomes, unsupported capabilities, and raw evidence paths, with the checked-in mlx-vlm Qwen3.8-27B-8bit baseline procedure documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai-infra-agents, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100