kubernetes-sigs / kubernetes-sigs/devops-bench
feat(results): Standardize universal token accounting, rejudge workflows, and offline results normalization
- Dominant language
- Python
- Stars
- 29
- Forks
- 15
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 15
Description
### Background & Problem Statement
Evaluating models across evaluation runs produces complex result artifacts. Inconsistencies in token parsing, evaluation re-judging, and result normalization currently cause evaluation discrepancies:
1. **Token Accounting Inconsistencies**: CLI harnesses serialize token metrics differently (e.g. camelCase `promptTokens` / `cachedContentTokenCount` vs snake_case), leading to dropped cached-token statistics in results aggregation.
2. **Re-judge History Overwrites**: When re-evaluating historical runs with updated rubrics or judge models, previous judge outputs were overwritten without a persistent backup prefix.
3. **Historical Data Re-normalization**: Schema adjustments and metric bug fixes require offline repair tooling to rebuild `rows.json` from raw trajectory logs without re-running expensive benchmark evaluations.
4. **Credential Leakage in Artifacts**: Result archives previously bundled live API keys, session tokens, and kubeconfig certificates.
### Proposed Architecture & Scope
1. **Universal Token Extraction & Serialization (`devops_bench/results/`)**:
- Normalize camelCase, snake_case, and nested token dictionaries across all agent harness outputs (cached tokens, input tokens, output/reasoning tokens).
2. **Deterministic Re-Judge Workflows (`scripts/rescore/`)**:
- `rejudge_runs.py --check --write`: Re-judges historical runs with fail-closed score withholding and stores original judgments under `rejudge-backups/`.
3. **Offline Results Re-normalization Tooling (`tools/renormalize-rows.py`)**:
- Provide a CLI tool to rebuild and validate `rows.json` directly from archived evaluation run directories across single and matrix execution layouts.
4. **Universal Results Data Loader (`devops_bench/results/loader.py`)**:
- Standardize data loader supporting different output structures and deduplicating by run ID.
5. **Length-Preserving Credential Scrubbing (`scripts/curation/`)**:
- `scripts/scrub_credentials.py`: Binary-safe, length-preserving secret scrubbing tool to purge API keys, cloud tokens, and certificates from transcripts and database files before archiving.
### Acceptance Criteria
- [ ] Token usage metrics (prompt tokens, cached tokens, completion tokens) parse consistently across all harnesses into `results.json` and `rows.json`.
- [ ] Re-judging a run preserves a timestamped backup of the original judgment record.
- [ ] `renormalize_rows.py` successfully recovers and normalizes evaluation directories into valid `rows.json` files.
- [ ] `scrub_credentials.py` removes all sensitive tokens and certificates while preserving SQLite transcript database integrity.
Contributor guide
Research direction
Read the entry points in devops_bench/results/, scripts/rescore/rejudge_runs.py, tools/renormalize-rows.py, and scripts/scrub_credentials.py to map the separate normalization, re-judge, offline repair, and scrubbing workflows. Start by tracing the existing result and archive layouts, then inspect the named CLI modes such as --check --write. Done means the acceptance criteria hold across harness outputs and archived runs without exposing credentials or corrupting SQLite transcripts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, devops, security, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100