kubernetes-sigs / kubernetes-sigs/devops-bench

feat(results): Standardize universal token accounting, rejudge workflows, and offline results normalization

Open
#169 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

### Background & Problem Statement
Evaluating models across evaluation runs produces complex result artifacts. Inconsistencies in token parsing, evaluation re-judging, and result normalization currently cause evaluation discrepancies:
1. **Token Accounting Inconsistencies**: CLI harnesses serialize token metrics differently (e.g. camelCase `promptTokens` / `cachedContentTokenCount` vs snake_case), leading to dropped cached-token statistics in results aggregation.
2. **Re-judge History Overwrites**: When re-evaluating historical runs with updated rubrics or judge models, previous judge outputs were overwritten without a persistent backup prefix.
3. **Historical Data Re-normalization**: Schema adjustments and metric bug fixes require offline repair tooling to rebuild `rows.json` from raw trajectory logs without re-running expensive benchmark evaluations.
4. **Credential Leakage in Artifacts**: Result archives previously bundled live API keys, session tokens, and kubeconfig certificates.

### Proposed Architecture & Scope
1. **Universal Token Extraction & Serialization (`devops_bench/results/`)**:
- Normalize camelCase, snake_case, and nested token dictionaries across all agent harness outputs (cached tokens, input tokens, output/reasoning tokens).
2. **Deterministic Re-Judge Workflows (`scripts/rescore/`)**:
- `rejudge_runs.py --check --write`: Re-judges historical runs with fail-closed score withholding and stores original judgments under `rejudge-backups/`.
3. **Offline Results Re-normalization Tooling (`tools/renormalize-rows.py`)**:
- Provide a CLI tool to rebuild and validate `rows.json` directly from archived evaluation run directories across single and matrix execution layouts.
4. **Universal Results Data Loader (`devops_bench/results/loader.py`)**:
- Standardize data loader supporting different output structures and deduplicating by run ID.
5. **Length-Preserving Credential Scrubbing (`scripts/curation/`)**:
- `scripts/scrub_credentials.py`: Binary-safe, length-preserving secret scrubbing tool to purge API keys, cloud tokens, and certificates from transcripts and database files before archiving.

### Acceptance Criteria
- [ ] Token usage metrics (prompt tokens, cached tokens, completion tokens) parse consistently across all harnesses into `results.json` and `rows.json`.
- [ ] Re-judging a run preserves a timestamped backup of the original judgment record.
- [ ] `renormalize_rows.py` successfully recovers and normalizes evaluation directories into valid `rows.json` files.
- [ ] `scrub_credentials.py` removes all sensitive tokens and certificates while preserving SQLite transcript database integrity.

Contributor guide

Open the contributing guide

Research direction

Read the entry points in devops_bench/results/, scripts/rescore/rejudge_runs.py, tools/renormalize-rows.py, and scripts/scrub_credentials.py to map the separate normalization, re-judge, offline repair, and scrubbing workflows. Start by tracing the existing result and archive layouts, then inspect the named CLI modes such as --check --write. Done means the acceptance criteria hold across harness outputs and archived runs without exposing credentials or corrupting SQLite transcripts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, devops, security, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.