google-deepmind / google-deepmind/eval_hub
Proposal: metamorphic robustness evaluation for reasoning benchmarks
- Dominant language
- Python
- Stars
- 28
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
## Proposal
Add a reusable **metamorphic evaluation** layer to `eval_hub` for testing whether a model remains correct under semantics-preserving transformations of the same underlying problem.
Instead of scoring only one canonical prompt, an evaluation item could optionally define a small family of equivalent variants, for example:
- rename variables or entities without changing the problem;
- reorder independent premises or irrelevant context;
- change formatting or presentation while preserving semantics;
- apply algebraically equivalent rewrites to mathematical expressions;
- paraphrase instructions while preserving the required answer contract.
The evaluator would score both ordinary correctness and **consistency across equivalent variants**.
## Motivation
A model can obtain a high benchmark score while still being brittle to superficial changes in wording, ordering, notation, or formatting. For reasoning evaluations, that makes it difficult to distinguish robust problem solving from dependence on one particular surface form.
A paired metamorphic score would expose this directly. For each base item, the runner could report:
- base accuracy;
- transformed accuracy;
- consistency rate across variants;
- cases where the answer changes even though the underlying problem does not.
This could be introduced without changing existing released evaluations. A benchmark would opt in by supplying transformations and a relation saying what must remain invariant.
A first implementation could be deliberately small and deterministic, with transformations authored or validated by humans rather than generated dynamically. `putnam_like` would be a natural demonstration because variable renaming, equivalent algebraic forms, and presentation changes can be defined precisely.
## Possible interface
Conceptually:
```python
MetamorphicCase(
base=problem,
variants=[variant_a, variant_b, variant_c],
relation="same_answer",
)
```
or a benchmark-specific transformation function returning equivalent prompt variants plus their expected answer relation.
The runner would preserve ordinary per-sample results while adding grouped robustness metrics, so existing evaluation outputs remain usable.
## Scope
Initial scope:
- deterministic, checked-in transformations;
- exact or benchmark-defined equivalence relations;
- grouped reporting for base/variant accuracy and consistency;
- one example integration, preferably `putnam_like`;
- no model-generated paraphrases in the trusted evaluation path.
If this direction is useful, I would be happy to prototype the common data model, runner/reporting support, tests, and one concrete evaluation integration.
I searched current issues and PRs for metamorphic, perturbation, paraphrase, invariance, and robustness evaluation and did not find an equivalent proposal.
Contributor guide
Research direction
Start by reading the existing eval_hub evaluation runner and reporting paths, then inspect the putnam_like evaluation as the proposed demonstration. Trace how per-sample results and benchmark integrations are represented before defining the common metamorphic data model. Done means deterministic checked-in transformations, grouped accuracy and consistency metrics, preserved existing outputs, tests, and one putnam_like integration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100