google-deepmind / google-deepmind/eval_hub

Putnam grade_samples reruns do not skip previously graded samples

Open Beginner friendly
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
6
PR merge metrics
No merged PRs in 30d

Description

## Summary

`eval_hub/putnam_like/grade_samples.py` reports a `SKIPPED (already graded)` state, but its skip check only looks for a grade filename containing the timestamp of the current grading run.

Because `grader_timestamp` changes on every invocation, a sample graded in an earlier run is not recognized as already graded and is sent to the grader again.

## Reproduction

`main()` creates a new timestamp for each run:

```python
grader_timestamp = time.strftime("%Y%m%d-%H%M%S")
```

`grade_gemini_sample()` then constructs:

```python
output_filename = (
f"grade_{FLAGS.model_name}_{grader_timestamp}_{sample_path.stem}.json"
)
```

and only skips when that exact path already exists.

For example, if a previous run produced:

```text
grade_gemini-2.5-pro_20260101-000000_sample_001.json
```

a later run with timestamp `20260822-100000` checks only for:

```text
grade_gemini-2.5-pro_20260822-100000_sample_001.json
```

and therefore re-grades the same sample.

## Impact

- repeated runs can issue unnecessary grading API calls and incur additional cost;
- the same sample can accumulate duplicate grades from the same grader model;
- the final report's `Samples Skipped (already graded)` behavior does not work across separate runs.

## Proposed fix

Before grading, detect an existing grade for the same grader model and sample stem regardless of its run timestamp, while still allowing a different grader model to grade the sample.

A focused regression test can create a prior-run grade file and verify that `grade_gemini_sample()` returns `SKIPPED` when invoked with a different current timestamp.

Contributor guide

Open the contributing guide

Research direction

Start in eval_hub/putnam_like/grade_samples.py by reading grade_gemini_sample() and the timestamp-based output filename check. Add a focused regression test that creates a prior-run grade file and invokes the function with a different timestamp. Done means the same grader model and sample are skipped across runs, while a different grader model can still grade them.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.