Farama-Foundation / Farama-Foundation/MiniWoB-plusplus

[Bug Report] `find-greatest` gives a positive terminal reward after submitting a non-maximum card

Open
#108 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
401
Forks
59
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

`find-greatest` asks the user/agent to pick the card with the greatest number and then submit:

```text
Find and pick the card with the greatest number, then press submit.
```

However, the task source gives a positive terminal reward when the selected card is not the greatest card:

```js
if(userIndex === expectedIndex.toString()) core.endEpisode(1.0, true);
else core.endEpisode(0.1, true);
```

This means a wrong final answer can end the episode with positive reward.

In a concrete downstream run, the page had card values `4`, `7`, and `2`. The agent selected `2` and submitted. The final DOM still showed another card with value `7`, so the selected card was not the greatest card. The run nevertheless received `RAW_REWARD_GLOBAL=0.1`. In downstream wrappers that treat positive raw reward as success, such as BrowserGym's MiniWoB wrapper, this becomes a successful run.

If the positive `0.1` reward is intentional shaping, then it would be helpful to clarify that positive terminal reward does not imply task success for this environment. Otherwise, wrong-card submission should probably return `0` or a negative reward rather than a positive terminal reward.

**Code example**

The relevant task code in `miniwob/html/miniwob/find-greatest.html` is:

```js
d3.select('button').on('click', function(){
if(d3.selectAll('.card.hidden')[0].length === 3) {core.endEpisode(-1.0, false); return;}
var userIndex = d3.select('.card:not(.hidden)')[0][0].getAttribute('data-index');
if(userIndex === expectedIndex.toString()) core.endEpisode(1.0, true);
else core.endEpisode(0.1, true);
});
```

A minimal manual reproduction is:

1. Open `find-greatest.html`.
2. Reveal/select a card that is not the greatest number.
3. Press `Submit`.
4. The wrong-card branch ends the episode with reward `0.1`.

The attached evidence package includes one concrete run showing the downstream effect:

- `evidence/final_dom_step_002_agent_a.html`: final DOM has cards `4`, `7`, and selected `2`.
- `evidence/native_evaluator_output_agent_a.json`: reports `RAW_REWARD_GLOBAL=0.1`, downstream `reward=1.0`, and `success=true`.
- `source/find-greatest.html`: local copy of the task source with the wrong-card `core.endEpisode(0.1, true)` branch.

For context, the downstream BrowserGym wrapper currently uses:

```python
reward = float(info["RAW_REWARD_GLOBAL"] > 0)
```

so the MiniWoB++ wrong-card reward `0.1` is interpreted as success there.

**System Info**

* MiniWoB++ installation: observed from a BrowserGym MiniWoB run using a bundled/local MiniWoB++ source snapshot, not from a direct `pip install miniwob` run. The attached package includes the exact `find-greatest.html` source snapshot used for the run.
* OS: macOS 26.2, Darwin 25.2.0, arm64.
* Python version: Python 3.14.3 on the machine used to inspect/package the evidence. The archived run itself was produced through the BrowserGym MiniWoB integration.

**Additional context**

This may be partly a downstream interpretation issue: BrowserGym treats any positive MiniWoB raw reward as success. Still, the root trigger is that `find-greatest` emits a positive terminal reward for a wrong submitted card. That is surprising for a task whose instruction asks for the greatest card, because a wrong submitted final answer can look like a successful completion to downstream consumers.

A possible fix in MiniWoB++ would be to make wrong-card submission return `0` or a negative reward. If the `0.1` partial reward is intentional, the documentation or environment metadata should make clear that positive terminal reward is not equivalent to task success for this task.

### Checklist

- [x] I have checked that there is no similar [issue](https://github.com/Farama-Foundation/miniwob-plusplus/issues) in the repo (**required**)

Contributor guide

Open the contributing guide

Research direction

Read miniwob/html/miniwob/find-greatest.html and inspect the submit handler's wrong-card branch. Reproduce the behavior by selecting a non-greatest card and submitting, then decide whether the intended contract is a non-positive reward or clarified partial-reward semantics. Done means the task behavior or documentation no longer lets downstream consumers mistake a wrong submission for success.

Written by the indexing model from the issue text.

Assessment

Tech stack
html, javascript
Domain
testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.