Farama-Foundation / Farama-Foundation/Arcade-Learning-Environment

ALE v0.6 differences in start state

Open
#291 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.5k
Forks
477
Avg merge
11h 11m
Merged PRs (30d)
3

Description

There was an issue raised (https://github.com/openai/gym/issues/1777) which describes differences between `v0.5.2` and `v0.6.0` of the ALE. I traced some of the issues to this commit https://github.com/mgbellemare/Arcade-Learning-Environment/commit/7bff96b4b64edcffbeb2d9bb83b1685ab506ea2b#diff-d9d868097a7403416e6ef352d95dc4feR178 which changes how `StellaEnvironment::softReset` works.

The `RESET` action is called `m_num_reset` times which leads to a different starting state for the agent. Perhaps this was intended behaviour in `StellaEnvironment::reset` but has ill-intended consequences in `StellaEnvironment::softReset`.

For example, here are the starting states for Ms. Pacman in ALE `v0.5.2` and `v0.6.0`. Note if you emulate **one** `RESET` action then we get the `v0.5.2` starting state.

### Ms. Pacman, ALE v0.5.2
![frame-v0 5 2](https://user-images.githubusercontent.com/1377567/72210487-8ed17400-3479-11ea-8705-fc0298fb9461.png)

### Ms. Pacman, ALE v0.6.0
![frame-v0 6 0](https://user-images.githubusercontent.com/1377567/72210490-96911880-3479-11ea-8cc5-52cfa077c01b.png)

You can see the subtle changes between these two frames (e.g., the colour of ghosts in jail).

I haven't looked into why we repetitively call `RESET`. Should this be something that is investigated further? It wouldn't seem that this should affect asymptotic performance.

Contributor guide

Open the contributing guide

Research direction

Start with StellaEnvironment::reset and StellaEnvironment::softReset, then inspect the linked ALE commit and the RESET action handling. Compare the reset counts and Ms. Pacman starting states between ALE v0.5.2 and v0.6.0; done means the intended soft-reset behavior is established and the version difference is resolved or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
game-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.