PolicyEngine / PolicyEngine/policybench

Restore public cost reporting with cumulative, comparable accounting

Open
#118 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
3
Avg merge
12h 7m
Merged PRs (30d)
13

Description

Context

The public leaderboard cost column is being hidden because the current values are not consistently comparable across models:

  • costs mix provider-reported, reconstructed, fallback-priced, and carried aggregate values;
  • actual Batch spend and synchronous-equivalent estimates are not always separated;
  • models use different validated serving recipes and output chunk sizes; and
  • external response retries and row-repair merges can replace prior usage rows instead of accumulating every attempted request, so some displayed totals can omit real spend.

The underlying cost fields and benchmark artifacts remain available internally. This issue tracks the work required before restoring a public cost surface.

Required work

  • Add a run-level cumulative request ledger that records every attempt, including failed requests, full-response retries, and row repairs.
  • Record actual billed cost separately from standardized synchronous-equivalent cost.
  • Serialize cost provenance: provider or estimator, pricing snapshot date and source, pricing mode, estimate flag, input/output/reasoning tokens, cache reads/writes, request count, transport (sync or Batch), response contract, and outputs per request.
  • Decide and document the public metric: operational cost under each model's validated recipe, identical-request intrinsic efficiency, or clearly labeled separate metrics.
  • Re-onboard GPT-5.5 after #101 using the whole-scenario tool contract and 16,384 completion-token cap. Run a two-household smoke test, then rerun all 100 households if it converges. If whole-scenario serving fails, document and validate the smallest convergent chunk size.
  • Reconcile or explicitly label aggregate-only Fable cost provenance, the Grok fallback estimate, and Sonnet 5 standard versus introductory pricing.
  • Add a validator that rejects incomplete or internally inconsistent public cost records.
  • Add UI regression coverage for the chosen labels, provenance, and unavailable-cost states.

Acceptance criteria

  • Every displayed model has an explicit cost basis, serving recipe, pricing snapshot, and provenance.
  • Cumulative run totals reconcile to the complete request ledger, including attempts later replaced by retries or repairs.
  • Actual billed costs and standardized estimates are never conflated.
  • The GPT-5.5 rerun is complete, audited, and incorporated.
  • The public UI defines the metric precisely and only restores cost after all current models pass validation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the hidden public leaderboard cost column and the underlying cost fields and benchmark artifacts; trace how retries, row repairs, and aggregate values are represented. Define the public metric and provenance requirements before implementing the cumulative ledger and validator. Done means validated records reconcile all attempts, the GPT-5.5 rerun is audited, and the UI restores only clearly labeled costs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data, frontend, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.