openai / openai/monitorability-evals

Intervention bootstrap can omit reporting groups from individual replicates

Open
#54 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
99
Forks
17
PR merge metrics
No merged PRs in 30d

Description

Summary

bootstrapped_gmean_metric() samples all unique instances globally in the outer bootstrap and only groups by final_groups after resampling.

When final_groups contains multiple reporting groups, a bootstrap replicate can therefore omit a small group entirely. For example, with one instance in dataset A and 99 in dataset B, a size-100 global bootstrap omits A in roughly 37% of replicates. Those missing replicates are then absent from A's bootstrap rows, so A's final mean/std are computed only from the subset of replicates in which A happened to be sampled.

That conflicts with the documented/reporting contract that each reported group receives a bootstrap estimate for each replicate, and it differs from the scaffold's process/outcome bootstrap helper, which resamples instances within each dataset.

Expected behavior

Outer bootstrap sampling should preserve each reporting group and resample instances with replacement within final_groups, so every non-empty reporting group contributes one estimate per bootstrap replicate.

Suggested fix

Build the outer instance table with the reporting labels needed for final_groups, sample each reporting group independently while preserving its instance count, then continue the existing cross-fit and inner-bootstrap pipeline. Add a regression test with a highly imbalanced two-group input that verifies every bootstrap index contains both reporting groups.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at bootstrapped_gmean_metric() and compare its outer sampling with the scaffold’s process/outcome bootstrap helper, which resamples within each dataset. Add a regression test using a highly imbalanced two-group input, then verify that every bootstrap index contains both reporting groups and that the existing cross-fit and inner-bootstrap pipeline still runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.