aws-samples / aws-samples/sample-agent-cost-bench
avg_cost_usd / avg_credits divide by metered runs, not run count (avg × runs ≠ total)
- Dominant language
- Python
- Stars
- 74
- Forks
- 19
- Avg merge
- 30m
- Merged PRs (30d)
- 2
Description
## Summary
`BenchmarkRun.cost_stats_by_target()` computes `avg_cost_usd` and `avg_credits` by dividing by the number of runs that carry a cost/credit value, rather than by the run count `n`. If any run in a target's set reports no usage data, the reported per-run average is inflated and `avg_cost_usd × runs` no longer reconciles with `total_cost_usd`.
The JSON key is `avg_cost_usd` and the console column is `Avg Cost/Run` — both read as a mean over all runs.
## Where
`agent_cost_bench/models.py`, on `main` @ `72f7586`:
- L1055 — `avg_cost = total_cost / len(costed) if costed else 0.0`
- L1075 — `avg_credits = total_credits / len(credit_vals) if credit_vals else 0.0`
`n = len(results)` is already in scope. It is used for the neighbouring `avg_latency = sum(latencies) / n` and emitted as `"runs": float(n)` in the same dict — which is what suggests an oversight rather than a deliberate "mean over metered runs".
## Observed
A `model-compare` run over 10 tasks per target, where exactly one run per target recorded `cost_usd: null` / `total_credits: 0`:
| target | `total_cost_usd` | `runs` | reported `avg_cost_usd` | `total / runs` |
|---|---|---|---|---|
| claude-opus-4.8 | 2.1080 | 10 | 0.2342 (= total / 9) | 0.2108 |
| claude-sonnet-4.6 | 0.8164 | 10 | 0.0907 (= total / 9) | 0.0816 |
| claude-haiku-4.5 | 0.4164 | 10 | 0.0463 (= total / 9) | 0.0416 |
`summary.total_cost_usd` = 3.3408 is correct, but `Σ(reported avg × 10)` = 3.71 — about 11% high.
A `cli-compare` run of 9 tasks in the same session is **unaffected**, because there `len(costed) == n`. That coincidence is what isolates the denominator as the cause rather than the totals.
## Impact
The per-run figure is the headline cost number in the console table and the HTML report, and it silently fails to reconcile with the run total. Nothing in the output signals that some runs were unmetered, so there is no indication the denominator differs from `runs`.
## Suggested fix
Either:
1. **Divide by `n`**, so the average reconciles with the total (unmetered runs counting as zero), and expose the metered-run counts so partial metering is visible rather than silent; or
2. **Keep the current denominator but rename** the field (e.g. `avg_cost_usd_per_metered_run`) and surface `costed_runs` in the table and report.
I have a patch for option 1 — the two one-line changes plus `costed_runs` / `credited_runs` added to the stats dict. Inlining it here so it can be applied directly; happy to raise it as a PR instead if you'd prefer.
```diff
diff --git a/agent_cost_bench/models.py b/agent_cost_bench/models.py
--- a/agent_cost_bench/models.py
+++ b/agent_cost_bench/models.py
@@ -1052,7 +1052,7 @@ class BenchmarkRun:
costed = [r for r in results if r.cost_usd is not None]
total_cost = sum(r.cost_usd for r in costed)
passed_cost = sum(r.cost_usd for r in passed if r.cost_usd is not None)
- avg_cost = total_cost / len(costed) if costed else 0.0
+ avg_cost = total_cost / n if n else 0.0
latencies = [
r.cli_reported_seconds if r.cli_reported_seconds else r.duration_seconds
for r in results
@@ -1072,7 +1072,7 @@ class BenchmarkRun:
passed_credits = sum(
(r.native_credits or 0.0) for r in passed if r.native_credits is not None
)
- avg_credits = total_credits / len(credit_vals) if credit_vals else 0.0
+ avg_credits = total_credits / n if n else 0.0
out[target] = {
"runs": float(n),
@@ -1089,6 +1089,8 @@ class BenchmarkRun:
"credits_per_pass": (passed_credits / len(passed)) if passed else float("inf"),
"has_credits": 1.0 if credit_vals else 0.0,
"has_cost": 1.0 if costed else 0.0,
+ "costed_runs": float(len(costed)),
+ "credited_runs": float(len(credit_vals)),
}
return out
```
## Secondary, lower priority
`cost_per_pass` is `passed_cost / len(passed)`, i.e. the mean cost of the *passing* runs. The column label `Cost/Success` reads more naturally as total spend ÷ successful tasks, which is a different number whenever failures cost differently than passes. Worth documenting the definition in the README or renaming the column, since the two are easy to conflate when quoting results.
Contributor guide
Research direction
Start in agent_cost_bench/models.py at BenchmarkRun.cost_stats_by_target(), especially the lines calculating avg_cost and avg_credits. Compare those denominators with n and the emitted runs value, then verify that the chosen fix makes the averages reconcile with totals and makes partial metering visible if the added counts are retained.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- analytics
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100