openai / openai/monitorability-evals

Output cleaning coerces boolean metric fields to integer 0/1

Open Beginner friendly
#51 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
98
Forks
17
PR merge metrics
No merged PRs in 30d

Description

Summary

The scaffold's _clean() helper checks integer types before preserving booleans. In Python, bool is a subclass of int, so ordinary boolean metric fields are converted to 1 and 0 when dataframe records are prepared for JSON output.

This affects intervention per-instance fields such as eligible and eligible_wald_sel, which are boolean conditions in the metric API but can be serialized as numeric values in summary.json.

The numeric representation is truthy-equivalent, but it changes the output schema and makes audit/consumer code less reliable because a logical eligibility flag is indistinguishable by type from an integer metric or count.

Expected behavior

Boolean and NumPy boolean values should remain JSON booleans (true/false). Integer and NumPy integer values should continue to serialize as ordinary Python integers.

Suggested fix

Handle (bool, np.bool_) before the integer conversion in _clean() and add focused regression coverage for scalar cleaning and dataframe record conversion.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the scaffold's _clean() helper and trace the dataframe record conversion path. Add focused regression coverage for scalar cleaning and dataframe records, confirming Python and NumPy booleans remain JSON booleans while integer values remain ordinary Python integers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.