Studies API: support all efficiency metrics (SUE, Bayesian regret, utilitarian efficiency, ESUP, Condorcet efficiency)
- Dominant language
- Python
- Stars
- 8
- Forks
- 4
- Avg merge
- 8h 2m
- Merged PRs (30d)
- 2
Description
Follow-up to PR #52 (studies API). The runner there currently returns raw per-election tallies (elected candidate utility, utility winner utility, random baseline utility, Condorcet winner flag). We'd like a small metrics layer on top so users can request any of these five efficiency metrics directly, without hand-rolling the division/averaging themselves.
## The five metrics to support
1. **Social Utility Efficiency (SUE)** — ratio of the elected candidate's average utility to the utility winner's average utility (aggregated across elections).
2. **Bayesian Regret** — the inverted form: average regret of the elected candidate vs the utility winner, expressed relative to a random baseline. SUE and regret are linked (SUE = 1 - regret_method / regret_random). Lower regret is better.
3. **Utilitarian Efficiency (probability)** — the fraction of elections that have a utility winner in which the method elects the utility winner. Same fundamental metric appears in Green-Armytage, Tideman & Cosman (2016) and Lehtinen (2007); treat it as one metric.
4. **Expected Social Utility Performance (ESUP)** — Holliday & Pacuit (2025). Averages the per-election normalized utility: compute utility(winner)/utility(best) inside each election, then average those ratios. Mathematically different from classic SUE (which divides aggregate averages) — this is a genuinely distinct number.
5. **Condorcet Efficiency** — fraction of elections that have a Condorcet winner in which the method elects the Condorcet winner.
(Explicitly NOT in scope: SUE under strategic candidate repositioning — Robinette 2023. Adds extra modeling steps, defer for now.)
## Proposed API design
- User declares the list of metrics they want (e.g. `metrics=["sue", "bayesian_regret"]`) and the runner computes **only the tallies those metrics need**.
- Utility-based metrics (1–4) need per-election: elected utility, utility winner utility, random baseline utility, and a flag for whether the elected candidate IS the utility winner (one comparison — free once the utility sums exist).
- Condorcet efficiency additionally needs the pairwise comparison pass and the elected-==-Condorcet-winner flag, which is the only expensive part.
- So asking for only utility metrics behaves exactly like the old example scripts (single test, no Condorcet overhead); asking for both runs both. No separate on/off flag to learn.
- Keep the raw tallies available too, for users who want custom aggregations.
- Output: a tidy table of scenarios × metrics, plus whatever form the existing runner returns.
## Notes for whoever picks this up
- PR #52 is the base this builds on; it was authored by a Cursor agent and the subscription is expired, so treat it as code to review/extend, not a bot that will respond.
- Verify against the values in the existing example scripts (they should be unchanged when only the same metric is requested).
- References: Holliday & Pacuit 2025 "expected social utility performance"; Green-Armytage, Tideman & Cosman 2016; Lehtinen 2007 (both describe the utilitarian-efficiency probability).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing PR #52's studies API and the existing example scripts. Trace how the runner produces raw per-election tallies and how scenarios are returned. Done means requested metrics compute only their needed tallies, raw results remain available, the output includes scenarios and metrics, and existing example values remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100