coder / coder/balatrollm

bug: leaderboard runs affected by two balatrobot defects — economy short-collected, losses recorded as wins

Open
#85 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
106
Forks
11
PR merge metrics
No merged PRs in 30d

Description

## Description

We replayed the full BalatroBench CC0 dataset (241 runs, 18k+ actions) on a
vanilla-faithful Balatro simulator as a differential check. Two balatrobot-level
defects we found and filed upstream reach published results:

1. **cash_out race (coder/balatrobot#231).** `bot.py` calls `cash_out` the moment
it sees `ROUND_EVAL`; the endpoint fires `G.FUNCS.cash_out` before the
round-eval rows finish committing, so affected cash-outs collect one blind
reward short ($3/$4/$5), or $0 when the payout is only the reward row. In our
replay, of the runs with money divergence, 61 had live short (the race
direction) and 13 the opposite (unexplained, possibly a second mechanism).
Several dataset runs die in the shop on purchases they could have afforded
with the missing rewards.

2. **won-flag cross-run leak (coder/balatrobot#232).** The executor starts its
Balatro instances once and cycles the whole task list through them; each run
begins with `menu` + `start`, and `G.GAME.won` survives that path. Since
`bot.py` records `finish_reason = "won"` whenever `gamestate["won"]` is
truthy, a *loss* that occurs on an instance after that instance has produced
any win can be recorded — and published — as a **win**. We observed the
leaked flag 16 times in our own long-lived-instance campaigns; the exposure
here grows with tasks-per-instance.

## A note for dataset consumers

Independent of the above: in `gamestates.jsonl`, ~1% of tool calls are
valid-but-unexecutable on live; they consume a response without emitting a
gamestate, which shifts every later index. Anyone aligning `gamestates[i]` with
action `i` needs to re-align at those events (we detected it when a "divergence"
decoded exactly to the *following* step's `rearrange` arguments).

## Suggested Fixes

- Pick up the balatrobot fixes when they land.
- Affected wins are detectable after the fact: a recorded win whose final
gamestate never beat the ante-8 boss (chips/ante inspection) is the leak.
Consider re-scoring or flagging affected runs in the published data.
- Meanwhile, for benchmark-grade integrity: one game process per run (avoids all
cross-run G.GAME leaks), and delay `cash_out` until the eval rows commit.

Happy to share our replay tooling/results for cross-checking.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.