google-deepmind / google-deepmind/eval_hub
MRCR v2 default scorer accepts certification hash after leading text
- Dominant language
- Python
- Stars
- 28
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
`eval_hub/mrcr_v2/run_evaluation.py` documents that the 12-character certification hash must be the first 12 characters of the stripped model output. The task prompt likewise says to prepend the hash and include no other text.
However, `mrcr_v2_metric()` currently uses `prediction.rfind(random_hash)` and scores the substring after the last occurrence of the hash. As a result, an output with arbitrary leading text before the hash can receive full credit.
This is also inconsistent with the MRCR V2 README, which says `run_evaluation.py` provides the default evaluation mode used for reported MRCR numbers as well as a more lenient version of the metric. The public script currently exposes only the lenient `rfind()` behavior.
## Reproduction
For a target of:
```text
AbCd1234EfGhexpected answer
```
this prediction currently receives a score of `1.0`:
```text
preface AbCd1234EfGhexpected answer
```
because the scorer finds the hash after `preface ` and compares only the suffix.
Under the documented default behavior, that prediction should score `0.0` because the certification hash is not at the beginning of the stripped output.
## Proposed fix
- make `mrcr_v2_metric()` enforce the documented hash-prefix requirement;
- preserve the existing `rfind()` behavior as an explicit lenient scorer;
- add regression tests for strict-prefix rejection, valid prefix acceptance, and lenient last-hash handling.
I have a small backwards-compatible patch prepared for this.
Contributor guide
Research direction
Start in eval_hub/mrcr_v2/run_evaluation.py and inspect mrcr_v2_metric(), then compare its behavior with the MRCR V2 README. Add regression tests covering strict-prefix rejection, valid prefix acceptance, and lenient last-hash handling; done means the documented default and explicit lenient behavior are both covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100