`rtk gain` cannot represent a regression: saturating_sub floors negative savings to zero
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 81.1k
- Forks
- 5.1k
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 35
Description
Summary
Tracker::record computes savings with saturating_sub on usize, so when RTK's output
is larger than the baseline the result is floored to 0. The schema has no
representation for "RTK made this worse": saved_tokens can never be negative, and no
aggregate — rtk gain, --daily, --weekly, --graph, --quota, --format json —
can surface a regression.
RTK does inflate some outputs, so these cases exist and are currently invisible.
Location
src/core/tracking.rs:425 (present on master @ b34be37 and in v0.45.0):
let saved = input_tokens.saturating_sub(output_tokens);
let pct = if input_tokens > 0 {
(saved as f64 / input_tokens as f64) * 100.0
} else {
0.0
};
Both operands come from estimate_tokens at src/core/tracking.rs:1394.
Evidence that regressions exist
Measured on a pinned Rust repository (457 .rs files), comparing raw output bytes against
rtk <cmd> output bytes:
| command | raw | rtk | change |
|---|---|---|---|
ls src |
251 | 341 | +35% |
ls -R src |
6,926 | 8,956 | +29% |
cargo check --workspace --all-targets |
72 | 100 | +38% |
Each of these records saved_tokens = 0, savings_pct = 0.0 — indistinguishable from a
command where RTK correctly declined to act.
Verified against the tracking DB directly:
$ sqlite3 ~/Library/Application\ Support/rtk/history.db \
"SELECT COUNT(*) FROM commands WHERE saved_tokens < 0;"
0
$ sqlite3 ~/Library/Application\ Support/rtk/history.db \
"SELECT COUNT(*) FROM commands WHERE output_tokens > input_tokens;"
0
Zero by construction, not by observation.
Why it matters
The aggregate savings number is the project's headline metric and the basis of the
--quota estimate. Flooring regressions to zero biases it upward: a deployment that is
net-negative for a given user cannot show that anywhere in the tooling. It also hides the
signal maintainers would most want — which handlers regress, and on what inputs.
Proposed fix
1. Store savings signed. Change saved_tokens to i64 and let savings_pct go
negative:
let saved = input_tokens as i64 - output_tokens as i64;
let pct = if input_tokens > 0 {
(saved as f64 / input_tokens as f64) * 100.0
} else {
0.0
};
Displays that assume non-negative values need a max(saved, 0) guard, or better, an
explicit branch so a loss renders as a loss. Existing rows remain valid — they are simply
non-negative.
2. Add a Regressions section to rtk gain, listing commands where output grew,
ordered by total tokens added. This turns an invisible failure into a work queue.
3. Optional, and arguably the real fix — a do-no-harm guard. run_filtered already
holds both the raw and filtered output at the point of decision. When the filtered form is
larger, emit the raw output and record the row as a no-op. That converts every regression
into a neutral outcome at negligible cost, and makes the guarantee "RTK never makes it
worse" true by construction rather than by handler-by-handler tuning.
I'd suggest (1) and (2) as one small change, with (3) as a follow-up since it changes
runtime behaviour rather than only accounting.
Environment
- rtk 0.45.0 (Homebrew), macOS arm64
- Confirmed still present on
master@b34be37
Related
- #2762 — reports
rtk gainsavings that are not reproducible from measured output.
This issue is one structural cause: the metric cannot express the negative half of the
distribution. - #561 —
lstracking baseline; the samelshandler produces two of the regressions above.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/core/tracking.rs at Tracker::record (line 425) and estimate_tokens (line 1394), then trace the rtk gain paths for --daily, --weekly, --graph, --quota, and --format json. Done means regressions are preserved as negative savings and surfaced by rtk gain, while existing non-negative rows remain valid; use the issue's sqlite queries to verify negative rows can be represented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sqlite
- Domain
- cli, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100