google-deepmind / google-deepmind/eval_hub
Putnam grade_samples reruns do not skip previously graded samples
- Dominant language
- Python
- Stars
- 28
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
`eval_hub/putnam_like/grade_samples.py` reports a `SKIPPED (already graded)` state, but its skip check only looks for a grade filename containing the timestamp of the current grading run.
Because `grader_timestamp` changes on every invocation, a sample graded in an earlier run is not recognized as already graded and is sent to the grader again.
## Reproduction
`main()` creates a new timestamp for each run:
```python
grader_timestamp = time.strftime("%Y%m%d-%H%M%S")
```
`grade_gemini_sample()` then constructs:
```python
output_filename = (
f"grade_{FLAGS.model_name}_{grader_timestamp}_{sample_path.stem}.json"
)
```
and only skips when that exact path already exists.
For example, if a previous run produced:
```text
grade_gemini-2.5-pro_20260101-000000_sample_001.json
```
a later run with timestamp `20260822-100000` checks only for:
```text
grade_gemini-2.5-pro_20260822-100000_sample_001.json
```
and therefore re-grades the same sample.
## Impact
- repeated runs can issue unnecessary grading API calls and incur additional cost;
- the same sample can accumulate duplicate grades from the same grader model;
- the final report's `Samples Skipped (already graded)` behavior does not work across separate runs.
## Proposed fix
Before grading, detect an existing grade for the same grader model and sample stem regardless of its run timestamp, while still allowing a different grader model to grade the sample.
A focused regression test can create a prior-run grade file and verify that `grade_gemini_sample()` returns `SKIPPED` when invoked with a different current timestamp.
Contributor guide
Research direction
Start in eval_hub/putnam_like/grade_samples.py by reading grade_gemini_sample() and the timestamp-based output filename check. Add a focused regression test that creates a prior-run grade file and invokes the function with a different timestamp. Done means the same grader model and sample are skipped across runs, while a different grader model can still grade them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100