automl / automl/Mighty

Support for Gymnasium 1.0

Open
#108 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
61
Forks
2
PR merge metrics
No merged PRs in 30d

Description

The project currently does not support the new gymnasium 1.0 API.

Among a few minor changes, the reset behavior of environments has changed.
Previously, the termination step returned the next_observation as the one from *after the reset*. To circumvent this, this observation currently gets overwritten using `info["final_observation"]`.
In the new API, the last observation correctly returns *the final observation* and the next `env.step` will return the "invalid" transition from before the reset to after the reset. **This step can not be used for learning, as it crosses reset boundaries.**
For more information look into the [Gymnasium Release Notes](https://gymnasium.farama.org/main/gymnasium_release_notes/) or [this short writeup](https://github.com/vwxyzjn/cleanrl/issues/499#issuecomment-2688317582) in the CleanRL-repo.

The `MightyAgent` class currently stores the replay buffer as a list of lists of e.g. returns. The second-level list signifies the different environments.
Since reset boundaries might be crossed by the environments at different points in time, we can not just throw away the transition using this structure.
We also can not keep these transitions, as they will lead to performance degradation.
A solution will probably require rewriting quite some code regarding the buffer to make this work, but I am not knowledgeable enough about this repository to properly gauge that.

There is a [backwards-compatible API](https://farama.org/Vector-Autoreset-Mode) using the Autoreset-mode "Same Step", but this will break some wrappers that are currently used.

I have some code lying around that lets the code run, but it currently does not discard the "faulty" trajectories.

- [ ] **Adapt code to new reset-api**
- [ ] (?) Restructure replay buffer
- [ ] Change some wrapper imports
- If compatibility with older versions of Gymnasium is desirable:
- [ ] Keep old and new code by conditionally running based on Gymnasium's version

Contributor guide

Open the contributing guide

Research direction

Start by reading the Gymnasium release notes and inspecting the MightyAgent replay-buffer code and wrapper imports mentioned in the issue. Adapt reset handling so transitions crossing reset boundaries are discarded, update the affected imports, and verify whether compatibility with older Gymnasium versions is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.