openai / openai/monitorability-evals

Bootstrap configuration accepts zero group sampling and fails after metric computation starts

Open Beginner friendly
#46 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
98
Forks
17
PR merge metrics
No merged PRs in 30d

Description

Summary

bootstrapped_gmean_metric() validates n_bootstrap and selection_frac, but does not validate BootstrapConfig.group_keep_frac before using it as DataFrame.sample(frac=...) for the outer bootstrap.

A zero group fraction therefore creates an empty bootstrap sample and lets execution continue into downstream merges/concatenation instead of failing at the configuration boundary with an actionable error. Negative values are delegated to pandas and surface implementation-specific errors. This makes an invalid evaluation configuration fail late and inconsistently.

Expected behavior

The metric should reject non-positive group_keep_frac values before starting any bootstrap work, just as it already rejects invalid n_bootstrap and selection_frac values.

Values greater than 1 can remain valid because sampling uses replace=True and can intentionally draw more bootstrap instances than the source set.

Suggested fix

Add a focused validation:

if bootstrap.group_keep_frac <= 0:
    raise ValueError(...)

and regression coverage for zero and negative fractions while preserving a group_keep_frac > 1 control.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at bootstrapped_gmean_metric() and review the existing validation for n_bootstrap and selection_frac alongside BootstrapConfig.group_keep_frac. Add regression coverage for zero and negative fractions, while preserving a group_keep_frac greater than 1 control. Done means invalid non-positive fractions fail at the configuration boundary before bootstrap work begins.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.