google-deepmind / google-deepmind/eval_hub

MRCR v2 default scorer accepts certification hash after leading text

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
6
PR merge metrics
No merged PRs in 30d

Description

## Summary

`eval_hub/mrcr_v2/run_evaluation.py` documents that the 12-character certification hash must be the first 12 characters of the stripped model output. The task prompt likewise says to prepend the hash and include no other text.

However, `mrcr_v2_metric()` currently uses `prediction.rfind(random_hash)` and scores the substring after the last occurrence of the hash. As a result, an output with arbitrary leading text before the hash can receive full credit.

This is also inconsistent with the MRCR V2 README, which says `run_evaluation.py` provides the default evaluation mode used for reported MRCR numbers as well as a more lenient version of the metric. The public script currently exposes only the lenient `rfind()` behavior.

## Reproduction

For a target of:

```text
AbCd1234EfGhexpected answer
```

this prediction currently receives a score of `1.0`:

```text
preface AbCd1234EfGhexpected answer
```

because the scorer finds the hash after `preface ` and compares only the suffix.

Under the documented default behavior, that prediction should score `0.0` because the certification hash is not at the beginning of the stripped output.

## Proposed fix

- make `mrcr_v2_metric()` enforce the documented hash-prefix requirement;
- preserve the existing `rfind()` behavior as an explicit lenient scorer;
- add regression tests for strict-prefix rejection, valid prefix acceptance, and lenient last-hash handling.

I have a small backwards-compatible patch prepared for this.

Contributor guide

Open the contributing guide

Research direction

Start in eval_hub/mrcr_v2/run_evaluation.py and inspect mrcr_v2_metric(), then compare its behavior with the MRCR V2 README. Add regression tests covering strict-prefix rejection, valid prefix acceptance, and lenient last-hash handling; done means the documented default and explicit lenient behavior are both covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.