githubnext / githubnext/gh-aw-cao

Ship a read-only token-efficiency dashboard MVP

Open
#11,829 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
1
Avg merge
49m
Merged PRs (30d)
837

Description

## Context

The token optimizer now has:

- a merged operational-value contract defining one frozen repository/workflow/experiment opportunity and correctness-first attainment; and
- a merged dashboard data contract defining source authority, evidence grain, stable identities, comparability, lifecycle states, SQL/IndexedDB parity, retention, privacy, and fail-closed behavior.

The current `optimization/dashboard.json` still focuses on package run conclusions, retained outputs, and mixed optimization attainment. It does not help an operator understand fleet token consumption or identify where token-efficiency investigation should begin.

The newly specified `token-efficiency-opportunities`, `token-efficiency-interventions`, and `token-efficiency-comparisons` sources do not yet have producers. This MVP must not synthesize those records from high AIC, aggregate tokens, or safe-output creation. It should use canonical evidence already available through `usage`, `runs`, `workflows`, `agent-smells`, `outcomes`, and `operational-values`, and present unavailable future evidence honestly.

Issue #11818 and its follow-on optimization outputs also show why the dashboard must account for optimization cost and recommendation churn. Recommendations for different target repositories are not globally duplicate work; overhead and supersession must be attributed only through an explicit frozen opportunity and intervention lineage.

## Goal

Replace the current generic optimization package page with a coherent, read-only Token Efficiency dashboard that lets an operator answer:

1. Where is measured AIC concentrated across repositories and workflows?
2. How are input, output, cache-read, cache-write, and reasoning tokens distributed without combining overlapping token classes?
3. Which workflows have evidence-backed efficiency signals worth investigating?
4. What proposed savings, gross realized savings, attributable optimization overhead, net realized savings, and recommendation churn are supported by current evidence?

This issue does not create or dispatch the token optimizer worker and does not publish recommendations.

## Dashboard intent

**Intent:** Help an operator prioritize evidence-backed token-efficiency investigation across enrolled repositories while keeping measured spend, diagnostic signals, proposed work, optimizer overhead, recommendation churn, and net realized value distinct.

Keep the package dashboard independently valid at `optimization/dashboard.json`. Do not add the package page directly to `dashboard/site/dashboard.json`.

## Page design

Create one phone-readable page with no more than four essential views. The first view must be the executive visual summary.

### 1. Fleet AIC trend — essential

A line chart over the active time window showing measured AIC by day, with series split by repository or workflow only when the chart remains legible.

- Source from invocation-grain `usage` through a declarative query.
- Aggregate AIC exactly once per invocation.
- Do not add run-summary AIC to invocation AIC.
- Make missing or incomplete usage evidence visible in the page copy/data state rather than rendering zero spend.

### 2. Highest-impact workflows — essential

A ranked table grouped by canonical repository and workflow identity showing:

- total measured AIC;
- distinct runs;
- invocation count;
- AIC per run when the denominator is valid;
- resolved model;
- latest observation and links.

Rank in Dashboard Language by total measured AIC. This is a prioritization view, not proof that spend is avoidable.

### 3. Token and efficiency diagnostics — essential

A compact table or chart that preserves separate measures for:

- input tokens;
- output tokens;
- cache-read tokens;
- cache-write tokens;
- reasoning tokens;
- evidence-backed `agent-smells` relevant to token efficiency.

Do not create a synthesized total-token measure. Do not infer `unused-tool-schema`, caching defects, retry loops, or other new opportunity kinds without the explicit producer evidence required by the data contract.

### 4. Optimization value and churn — essential

A table presenting existing optimization outcomes and operational-value observations without treating recommendations as realized savings.

Where authoritative evidence exists, show:

- recommendation disposition and explicit supersession lineage;
- proposed savings;
- gross realized savings;
- optimizer/auditor/verifier and linked replacement-run overhead;
- net realized savings and verified net gain;
- recommendation churn count and rate;
- maturity, evaluator definition, evidence cutoff, and evidence links.

Only an `applied` recommendation with completed implementation and complete matured comparable target evidence may report realized value. Superseded, outdated, duplicate, unapplied, failed-start, and rejected recommendations report no realized value. Exclude unrelated repositories, workflows, opportunities, and portfolio discovery from intervention overhead. Use explicit empty/unavailable messaging where canonical opportunity, intervention, comparison, disposition, or overhead evidence has not yet been produced.

Existing package activation/run-conclusion detail may remain only as a supplemental disclosure if it is still useful.

## Declarative query requirements

- Define all filtering, joining, grouping, aggregation, calculation, ranking, ordering, and pagination in Dashboard Language queries inside `optimization/dashboard.json`.
- Execute through the production data Web Worker/query boundary with an explicit abort-scoped active-page subscription.
- Do not add presenter callbacks, derived JavaScript sources, component filtering, main-thread business calculations, or test-only source synthesis.
- Join using canonical organization/repository/workflow/run fields; never join by display title alone.
- Preserve source availability, completeness, freshness, and provenance through derived queries.
- Fail closed when a declared source cannot be produced.

## Required contract updates

Update together:

- `optimization/dashboard.json` queries and page declaration;
- Dashboard Language source/field validation only if the MVP needs fields already specified but not yet wired;
- production worker query/source resolution needed for the selected existing sources;
- focused dashboard contract fixture(s);
- unit tests resolving every declared source through the production worker/query boundary;
- focused E2E coverage for the page’s loading, populated, empty, unavailable, and reactive-update states.

If implementation reveals that an authoritative source field is not currently normalized, update `specs/dashboard-data.md` and `specs/dashboard-gh-aw-jsonl-mapping.md` before changing adapters. Do not guess a mapping.

## Acceptance criteria

- The optimization package dashboard is centered on token efficiency rather than generic package bookkeeping.
- The page begins with a line chart that communicates fleet AIC trend without scrolling on a phone-sized viewport.
- The page contains no more than four essential views.
- Operators can identify measured spend concentration by repository and workflow.
- AIC and all five raw token classes remain separately named and correctly grained.
- High spend is not labeled as avoidable waste without evidence.
- Proposed savings, gross realized savings, attributable optimization overhead, net realized savings, and verified net gain are distinct.
- Recommendation disposition, explicit supersession lineage, and churn are visible when authoritative evidence exists.
- Unrelated portfolio work is not charged to an intervention.
- Missing future opportunity/intervention/comparison producers appear as empty or unavailable evidence, never fabricated rows or zero savings.
- Every view source resolves through the production Web Worker/query boundary and updates when canonical database observations change.
- Package dashboard validation, focused unit tests, and focused Playwright coverage pass.
- From `dashboard/site/`, run the relevant subset of `npm test`, `npm run test:e2e`, `npm run lint`, and `npm run typecheck`; validate the package dashboard with the repository’s existing Dashboard Language validator.

## Out of scope

- Creating `.github/workflows/optimization-token-optimizer.md`.
- Dispatching optimization work or writing to target repositories.
- Producing token-efficiency opportunity, intervention, or comparison observations.
- Computing verified savings before the corresponding matured evidence exists.
- Portfolio consolidation or control-plane efficiency recommendations.

Depends on #11761 and its merged implementation in #11800.

Contributor guide

Open the contributing guide

Research direction

Start with optimization/dashboard.json and the existing production Web Worker/query boundary, then review the dashboard contract fixtures and tests under dashboard/site/. Check specs/dashboard-data.md and specs/dashboard-gh-aw-jsonl-mapping.md if source fields are not normalized. Done means the four-view read-only page resolves canonical evidence, handles unavailable states, reacts to updates, and passes the listed validation, unit, E2E, lint, and typecheck commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
analytics, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.