google-deepmind / google-deepmind/concordia

Contest run provenance: making model drift detectable with per-run output fingerprinting

Open
#271 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
367
PR merge metrics
No merged PRs in 30d

Description

The [NeurIPS 2024 contest](https://www.cooperativeai.com/contests/concordia-2024) evaluated agents against shared scenarios with a live LLM backend. PR #265 is doing useful work restoring the evaluation pipeline — but there is a problem the pipeline restoration cannot fix on its own: the model that scored the original contest entries no longer exists in that form.

PR #265 acknowledges this explicitly in its "Assumptions Made" section:

> Model generation schemas like `gpt-4o` are living APIs. This pipeline implements exact original mechanics algorithmically, but variations tied directly to background conversational drift via newer API weights are to be expected compared to initial scores extracted natively in Q4 2024.

So even with the pipeline restored, a claim of "I reproduced the original scores" is currently unverifiable. Two runs producing the same aggregate Elo could have completely different per-scenario per-agent outcomes — one might genuinely reproduce the original; the other might be compensating for model drift with accidentally correct aggregate numbers.

**What's missing**: a tamper-evident fingerprint of exactly what each run produced, at the per-scenario per-agent level.

---

### A concrete proposal

`run.py` already writes `simulation_log_{game}_{scenario}.json` and returns a `focal_scores` dict per scenario. A small companion script could:

1. Read the JSON outputs from a completed run across all scenarios
2. Build a SHA-256 Merkle root over all per-scenario per-agent scores — a `content_hash`
3. Capture model name, timestamp, and command separately in a `bundle_hash` (same content, different identity)

Two runs producing identical aggregate Elo but differing on any individual scenario or agent → different `content_hash`. Model drift becomes detectable: if your `content_hash` matches the reference run, the reproduction is scientifically equivalent at the per-scenario level.

For practical use:
- Official contest runs emit a `content_hash` alongside published scores
- Teams claiming reproduction include their `content_hash` — a mismatch means drift was detected, not just suspected
- Future editions can require submitters to include a `content_hash` with their entry, making leaderboard rows independently verifiable

---

### Implementation

I've been building exactly this format for a different reproducibility project — [`valichord_attestation`](https://github.com/topeuph-ai/ValiChord/tree/main/valichord_attestation) (pip-installable, no infrastructure). The concordia adapter would be ~50 lines: read `run.py`'s JSON outputs across scenarios, call `build_bundle()`, print `content_hash`.

Does this seem like a useful direction? A few questions that would help me scope a concrete PR:

1. Do the original 2024 contest run logs still exist somewhere? If so, computing a reference `content_hash` from them would give future reproducers something concrete to match against.
2. Would this sit better as an addition to PR #265, or as a separate script alongside `run.py`?
3. Is there an official format for how per-scenario scores are aggregated into the final Elo — the Merkle root needs to cover the right level of granularity to be meaningful.

Happy to prototype this and share a draft regardless.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.