DLR-RM / DLR-RM/stable-baselines3

[Question] why self.env.reset doesn't return an obs variable in FireResetEnv wrapper reset function, while it does in EpisodicLifeEnv and NoopResetEnv wrapper reset function

Open
#2,256 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
13.8k
Forks
2.2k
Avg merge
1h 35m
Merged PRs (30d)
2

Description

### ❓ Question

In FireResetEnv wrapper in the file of stable-baselines3/stable_baselines3/common/atari_wrappers.py, I believe it is an issue that there is no obs return from self.env.reset(**kwargs)
The code is as in the below:

```
def reset(self, **kwargs) -> AtariResetReturn:
self.env.reset(**kwargs)
obs, _, terminated, truncated, _ = self.env.step(1)
if terminated or truncated:
self.env.reset(**kwargs)
obs, _, terminated, truncated, _ = self.env.step(2)
if terminated or truncated:
self.env.reset(**kwargs)
return obs, {}
```

I believe it should be modified to

```
def reset(self, **kwargs) -> AtariResetReturn:
self.env.reset(**kwargs)
obs, _, terminated, truncated, _ = self.env.step(1)
if terminated or truncated:
self.env.reset(**kwargs)
obs, _, terminated, truncated, _ = self.env.step(2)
if terminated or truncated:
obs, _ = self.env.reset(**kwargs)
return obs, {}

```
I have encountered an issue with GravitarNoFrameskip-v4 environment, when I run the command

`python train.py --algo dqn --env GravitarNoFrameskip-v4 --seed 37 --eval-freq -1 --n-timesteps 10000000 --tensorboard-log ./runs/`

on my computer, when it was running to the step 23374, when FireResetEnv reset execute

`obs, _, terminated, truncated, _ = self.env.step(2)`,

the environment terminated, and executed

```
if terminated or truncated:
self.env.reset(**kwargs)
```

the code will execute EpisodicLifeEnv reset() and NoopResetEnv reset(), however, **_the retruned obs variable from FireResetEnv reset is still obs before reset process_**,

`obs, _, terminated, truncated, _ = self.env.step(2)`

while I think it should be the obs after reset process.

```
if terminated or truncated:
obs, _ = self.env.reset(**kwargs)
```

Here is the datalog after I add some print lines:

```
EpisodicLifeEnv step(): lives: 4 -> 3, terminated: True, truncated: False
common/vec_env/dummy_vec_env.py: DummyVecEnv step_wait(): before env reset for env 0
FireResetEnv reset() called with kwargs: {'seed': None, 'options': None}
EpisodicLifeEnv reset() called with kwargs: {'seed': None, 'options': None}
EpisodicLifeEnv reset() end: lives: 3
EpisodicLifeEnv reset() end: obs shape: (210, 160, 3), obs.sum(): 3139748.00, obs.mean(): 31.148294
FireResetEnv step() called with action 1 after, terminated: False, truncated: False
FireResetEnv step() called with action 2 before, terminated: False, truncated: False
********** Monitor wrapper terminated: True, episode_num: 37, episode reward: 250.0, episode length: 2358, total steps: 96075 **********
EpisodicLifeEnv step(): lives: 3 -> 0, terminated: True, truncated: False
FireResetEnv step() called with action 2 after, terminated: True, truncated: False
EpisodicLifeEnv reset() called with kwargs: {'seed': None, 'options': None}
EpisodicLifeEnv performing full reset
NoopResetEnv reset() called with kwargs: {'seed': None, 'options': None}
********** Monitor reset() called
********** Monitor reset() with kwargs: {'seed': None, 'options': None} end
NoopResetEnv reset() end: performed 21 no-ops, obs shape: (210, 160, 3), obs.sum(): 211087
EpisodicLifeEnv reset() end: lives: 6
EpisodicLifeEnv reset() end: obs shape: (210, 160, 3), obs.sum(): 211087.00, obs.mean(): 2.094117
FireResetEnv reset() end
FireResetEnv reset() end: obs shape: (210, 160, 3), obs.sum(): 3122004.00, obs.mean(): 30.972262
```

### Checklist

- [x] I have checked that there is no similar [issue](https://github.com/DLR-RM/stable-baselines3/issues) in the repo
- [x] I have read the [documentation](https://stable-baselines3.readthedocs.io/en/master/)
- [x] If code there is, it is [minimal and working](https://github.com/DLR-RM/stable-baselines3/issues/982#issuecomment-1197044014)
- [x] If code there is, it is formatted using the [markdown code blocks](https://help.github.com/en/articles/creating-and-highlighting-code-blocks) for both code and stack traces.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.