rtk-ai / rtk-ai/rtk

`rtk gain` cannot represent a regression: saturating_sub floors negative savings to zero

Open
#3,702 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:cli bug priority:high
Dominant language
Rust
Stars
81.1k
Forks
5.1k
Avg merge
4d 21h
Merged PRs (30d)
35

Description

Summary

Tracker::record computes savings with saturating_sub on usize, so when RTK's output
is larger than the baseline the result is floored to 0. The schema has no
representation for "RTK made this worse": saved_tokens can never be negative, and no
aggregate — rtk gain, --daily, --weekly, --graph, --quota, --format json
can surface a regression.

RTK does inflate some outputs, so these cases exist and are currently invisible.

Location

src/core/tracking.rs:425 (present on master @ b34be37 and in v0.45.0):

let saved = input_tokens.saturating_sub(output_tokens);
let pct = if input_tokens > 0 {
    (saved as f64 / input_tokens as f64) * 100.0
} else {
    0.0
};

Both operands come from estimate_tokens at src/core/tracking.rs:1394.

Evidence that regressions exist

Measured on a pinned Rust repository (457 .rs files), comparing raw output bytes against
rtk <cmd> output bytes:

command raw rtk change
ls src 251 341 +35%
ls -R src 6,926 8,956 +29%
cargo check --workspace --all-targets 72 100 +38%

Each of these records saved_tokens = 0, savings_pct = 0.0 — indistinguishable from a
command where RTK correctly declined to act.

Verified against the tracking DB directly:

$ sqlite3 ~/Library/Application\ Support/rtk/history.db \
    "SELECT COUNT(*) FROM commands WHERE saved_tokens < 0;"
0
$ sqlite3 ~/Library/Application\ Support/rtk/history.db \
    "SELECT COUNT(*) FROM commands WHERE output_tokens > input_tokens;"
0

Zero by construction, not by observation.

Why it matters

The aggregate savings number is the project's headline metric and the basis of the
--quota estimate. Flooring regressions to zero biases it upward: a deployment that is
net-negative for a given user cannot show that anywhere in the tooling. It also hides the
signal maintainers would most want — which handlers regress, and on what inputs.

Proposed fix

1. Store savings signed. Change saved_tokens to i64 and let savings_pct go
negative:

let saved = input_tokens as i64 - output_tokens as i64;
let pct = if input_tokens > 0 {
    (saved as f64 / input_tokens as f64) * 100.0
} else {
    0.0
};

Displays that assume non-negative values need a max(saved, 0) guard, or better, an
explicit branch so a loss renders as a loss. Existing rows remain valid — they are simply
non-negative.

2. Add a Regressions section to rtk gain, listing commands where output grew,
ordered by total tokens added. This turns an invisible failure into a work queue.

3. Optional, and arguably the real fix — a do-no-harm guard. run_filtered already
holds both the raw and filtered output at the point of decision. When the filtered form is
larger, emit the raw output and record the row as a no-op. That converts every regression
into a neutral outcome at negligible cost, and makes the guarantee "RTK never makes it
worse" true by construction rather than by handler-by-handler tuning.

I'd suggest (1) and (2) as one small change, with (3) as a follow-up since it changes
runtime behaviour rather than only accounting.

Environment

  • rtk 0.45.0 (Homebrew), macOS arm64
  • Confirmed still present on master @ b34be37

Related

  • #2762 — reports rtk gain savings that are not reproducible from measured output.
    This issue is one structural cause: the metric cannot express the negative half of the
    distribution.
  • #561 — ls tracking baseline; the same ls handler produces two of the regressions above.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/core/tracking.rs at Tracker::record (line 425) and estimate_tokens (line 1394), then trace the rtk gain paths for --daily, --weekly, --graph, --quota, and --format json. Done means regressions are preserved as negative savings and surfaced by rtk gain, while existing non-negative rows remain valid; use the issue's sqlite queries to verify negative rows can be represented.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite
Domain
cli, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.