Hmbown / Hmbown/Codewhale

No runtime performance gate exists: nothing measures or defends the speed of the user path

Open
#6,193 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
41k
Forks
3.6k
Avg merge
13h 59m
Merged PRs (30d)
299

Description

## What is missing

`BUILD_PERFORMANCE.md` covers *build* speed. Nothing in the repository measures
*runtime* speed: no `benches/`, no criterion / iai / divan in `Cargo.lock`, no
`.perf` cases, no budget constants, no gate that fails when a user path gets
slower. Verified 2026-09-15.

So every claim about responsiveness in this milestone ("streaming is bursty",
"typing feels laggy", "make it smooth and fast") is currently unfalsifiable. The
stream-pacing change in `9b41aeaac` was reproduced and measured with a scratch
harness that was thrown away; nothing keeps that measurement honest next month.

## What good looks like, learned from deepseek-harness

`refs/dsh/benchmarks/AGENTS.md` runs required repository-level performance gates
whose measured path crosses package ownership. The transferable parts, in the
order I would take them:

1. **Organize by measured user path, not by package tree.** One directory per
path. DSH has `session-open`, `agent-continuation`, `conversation-fold`,
`active-stream-reconnect`, `terminal-io`, `long-session-browser`.
2. **Fixed synthetic inputs from reviewed constants.** "Never use recorded
Sessions, user material, ambient repositories, or network services."
3. **Budgets are reviewed source constants.** "Environment variables must not
override performance budgets."
4. **Say how the verdict is reached.** "Report enough raw and aggregate
measurements to explain each verdict, including whether a budget uses a
median, minimum, absolute value, or ratio."
5. **Separate expectations by machine.** Reference-machine expectations are
recorded separately from the shared CI time scale and variance headroom, and
the time scale is never applied to memory or to dimensionless ratios.
6. **Fresh process, private root, bounded child** for any wall-clock or
retained-memory sample; remove owned roots after failure as well as success.
7. **One named benchmark per real defect class.** `active-stream-reconnect`
exists because a reconnect carrying a 100,000-delta reasoning prefix has to
fold without stalling — a specific shape, not a generic "streaming is fast".

## The decision this needs first

A harness means a benchmark dependency, and criterion / iai / divan are all
absent from `Cargo.lock`. Adding one is a dependency decision on a repository
currently working to *reduce* dependency surface (#6151), so it should be a
deliberate choice rather than a drive-by. The alternatives are a dependency-free
`Instant`-based harness in a child process, or adopting one of the above.

Also unanswered: what wall-clock assertion can survive a busy developer machine,
and which user paths are actually gated versus merely measured. DSH answers both;
we have not.

## First candidates, straight from this milestone

- **Stream reveal throughput and latency**: the per-beat advance, and the lag
between a provider delta arriving and its text being visible, at several
backlog sizes.
- **TUI input latency**: keystroke to painted frame.
- **Session open** on a long history — the TUI already virtualizes the
transcript (`4c8e5de`), so the cost is in the fold, not the render.

Not started. Filed because the milestone now contains speed claims that nothing
in the repository can check.

Contributor guide

Open the contributing guide

Research direction

Start with BUILD_PERFORMANCE.md and refs/dsh/benchmarks/AGENTS.md, then inspect Cargo.lock and the milestone paths named in the issue: stream reveal, TUI input, and session open. Compare the dependency-based and dependency-free harness options before defining which paths, budgets, measurements, and machine expectations are in scope. Done means the repository has documented, repeatable runtime measurements and gates that report verdicts and fail when approved budgets are exceeded.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.