Farama-Foundation / Farama-Foundation/MiniWoB-plusplus

[Ongoing] Find & mark non-deterministic environments

Open
#119 0 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
HTML
Stars
400
Forks
59
PR merge metrics
No merged PRs in 30d

Description

HTML and JavaScript are never meant to be run deterministically (or even synchronously), same story with web browsers. As a result, web-based MiniWoB++ suffer from determinism issues. Some are due to bugs in script, others are sort of inevitable.

If you find any non-deterministic environments, please share it via replying to this issue, and we would investigate & mark the environment as a non-deterministic one. It would be helpful to provide observation dumps or any other useful debug info, but just having the name is enough.

Relevant pull requests:
- #116
- #118

### Checklist

- [x] I have checked that there is no similar [issue](https://github.com/Farama-Foundation/miniwob-plusplus/issues) in the repo (**required**)

Contributor guide

Open the contributing guide

Research direction

Review the web-based MiniWoB++ environments and investigate whether their behavior is reproducible across runs. Share the environment name with observation dumps or other useful debug information; done means the issue has been investigated and the environment is marked as non-deterministic when appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
html, javascript
Domain
testing, web-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.