openai / openai/monitorability-evals
Bootstrap configuration accepts zero group sampling and fails after metric computation starts
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 98
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
Summary
bootstrapped_gmean_metric() validates n_bootstrap and selection_frac, but does not validate BootstrapConfig.group_keep_frac before using it as DataFrame.sample(frac=...) for the outer bootstrap.
A zero group fraction therefore creates an empty bootstrap sample and lets execution continue into downstream merges/concatenation instead of failing at the configuration boundary with an actionable error. Negative values are delegated to pandas and surface implementation-specific errors. This makes an invalid evaluation configuration fail late and inconsistently.
Expected behavior
The metric should reject non-positive group_keep_frac values before starting any bootstrap work, just as it already rejects invalid n_bootstrap and selection_frac values.
Values greater than 1 can remain valid because sampling uses replace=True and can intentionally draw more bootstrap instances than the source set.
Suggested fix
Add a focused validation:
if bootstrap.group_keep_frac <= 0:
raise ValueError(...)
and regression coverage for zero and negative fractions while preserving a group_keep_frac > 1 control.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at bootstrapped_gmean_metric() and review the existing validation for n_bootstrap and selection_frac alongside BootstrapConfig.group_keep_frac. Add regression coverage for zero and negative fractions, while preserving a group_keep_frac greater than 1 control. Done means invalid non-positive fractions fail at the configuration boundary before bootstrap work begins.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100