openai / openai/monitorability-evals
Intervention bootstrap can omit reporting groups from individual replicates
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 99
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
Summary
bootstrapped_gmean_metric() samples all unique instances globally in the outer bootstrap and only groups by final_groups after resampling.
When final_groups contains multiple reporting groups, a bootstrap replicate can therefore omit a small group entirely. For example, with one instance in dataset A and 99 in dataset B, a size-100 global bootstrap omits A in roughly 37% of replicates. Those missing replicates are then absent from A's bootstrap rows, so A's final mean/std are computed only from the subset of replicates in which A happened to be sampled.
That conflicts with the documented/reporting contract that each reported group receives a bootstrap estimate for each replicate, and it differs from the scaffold's process/outcome bootstrap helper, which resamples instances within each dataset.
Expected behavior
Outer bootstrap sampling should preserve each reporting group and resample instances with replacement within final_groups, so every non-empty reporting group contributes one estimate per bootstrap replicate.
Suggested fix
Build the outer instance table with the reporting labels needed for final_groups, sample each reporting group independently while preserving its instance count, then continue the existing cross-fit and inner-bootstrap pipeline. Add a regression test with a highly imbalanced two-group input that verifies every bootstrap index contains both reporting groups.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at bootstrapped_gmean_metric() and compare its outer sampling with the scaffold’s process/outcome bootstrap helper, which resamples within each dataset. Add a regression test using a highly imbalanced two-group input, then verify that every bootstrap index contains both reporting groups and that the existing cross-fit and inner-bootstrap pipeline still runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100