google-deepmind / google-deepmind/eval_hub

Proposal: metamorphic robustness evaluation for reasoning benchmarks

Open
#13 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
6
PR merge metrics
No merged PRs in 30d

Description

## Proposal

Add a reusable **metamorphic evaluation** layer to `eval_hub` for testing whether a model remains correct under semantics-preserving transformations of the same underlying problem.

Instead of scoring only one canonical prompt, an evaluation item could optionally define a small family of equivalent variants, for example:

- rename variables or entities without changing the problem;
- reorder independent premises or irrelevant context;
- change formatting or presentation while preserving semantics;
- apply algebraically equivalent rewrites to mathematical expressions;
- paraphrase instructions while preserving the required answer contract.

The evaluator would score both ordinary correctness and **consistency across equivalent variants**.

## Motivation

A model can obtain a high benchmark score while still being brittle to superficial changes in wording, ordering, notation, or formatting. For reasoning evaluations, that makes it difficult to distinguish robust problem solving from dependence on one particular surface form.

A paired metamorphic score would expose this directly. For each base item, the runner could report:

- base accuracy;
- transformed accuracy;
- consistency rate across variants;
- cases where the answer changes even though the underlying problem does not.

This could be introduced without changing existing released evaluations. A benchmark would opt in by supplying transformations and a relation saying what must remain invariant.

A first implementation could be deliberately small and deterministic, with transformations authored or validated by humans rather than generated dynamically. `putnam_like` would be a natural demonstration because variable renaming, equivalent algebraic forms, and presentation changes can be defined precisely.

## Possible interface

Conceptually:

```python
MetamorphicCase(
base=problem,
variants=[variant_a, variant_b, variant_c],
relation="same_answer",
)
```

or a benchmark-specific transformation function returning equivalent prompt variants plus their expected answer relation.

The runner would preserve ordinary per-sample results while adding grouped robustness metrics, so existing evaluation outputs remain usable.

## Scope

Initial scope:

- deterministic, checked-in transformations;
- exact or benchmark-defined equivalence relations;
- grouped reporting for base/variant accuracy and consistency;
- one example integration, preferably `putnam_like`;
- no model-generated paraphrases in the trusted evaluation path.

If this direction is useful, I would be happy to prototype the common data model, runner/reporting support, tests, and one concrete evaluation integration.

I searched current issues and PRs for metamorphic, perturbation, paraphrase, invariance, and robustness evaluation and did not find an equivalent proposal.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing eval_hub evaluation runner and reporting paths, then inspect the putnam_like evaluation as the proposed demonstration. Trace how per-sample results and benchmark integrations are represented before defining the common metamorphic data model. Done means deterministic checked-in transformations, grouped accuracy and consistency metrics, preserved existing outputs, tests, and one putnam_like integration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.