block / block/buzz

bench: qualify bleeding-edge local inference backends on real agent work

Open
#6,626 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Problem

Buzz can route agents to arbitrary OpenAI-compatible endpoints, and the Harbor Buzz Orchestra benchmark can swap endpoint deployment configuration. It does not yet produce a backend qualification receipt that makes rapidly changing local inference engines comparable and upstream-debuggable.

On Apple Silicon, backend/model changes can materially improve throughput while also regressing batching isolation, streaming tool calls, reasoning controls, cancellation, prefix caches, or long-running server health. Tokens per second alone is not a sufficient promotion gate for an agent workforce.

## Proposed first slice

Add a benchmark-only qualification path that:

- records exact backend, model, drafter, dependency, OS, hardware, and launch configuration provenance;
- exercises an OpenAI-compatible endpoint with focused correctness/concurrency probes before agent work;
- emits a machine-readable, append-only result receipt with raw evidence paths;
- keeps endpoint/backend selection outside Buzz product semantics;
- establishes a Qwen3.8-27B autoregressive MLX baseline before comparing MTP and other serving engines;
- can feed surviving conditions into the existing Harbor Buzz Orchestra workload harness.

## Initial acceptance criteria

- [ ] A local endpoint condition can be described without hard-coding MLX concepts into Buzz runtime code.
- [ ] The qualifier checks basic generation, streaming, reasoning controls, tool-call continuation, cancellation, concurrency isolation, and liveness after failures where supported.
- [ ] Unsupported capabilities are recorded explicitly rather than treated as passes.
- [ ] Every run emits a JSON receipt pinning revisions and configuration.
- [ ] Unit tests cover manifest validation and receipt serialization.
- [ ] The first checked-in example documents the source-pinned `mlx-vlm` autoregressive baseline procedure for `Qwen3.8-27B-8bit`.

## Non-goals

- Selecting a permanent inference engine.
- Adding MLX-specific concepts to the Buzz agent protocol.
- Publishing benchmark claims before repeated real-workload validation.
- Modifying or redistributing model weights.

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing Harbor Buzz Orchestra workload harness and the benchmark entry point that can describe endpoint conditions. Read how benchmark configuration and results are currently represented, then trace the requested manifest validation and receipt serialization tests. Done means a benchmark-only qualifier records provenance, probe outcomes, unsupported capabilities, and raw evidence paths, with the checked-in mlx-vlm Qwen3.8-27B-8bit baseline procedure documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai-infra-agents, performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.