DLR-RM / DLR-RM/stable-baselines3
[Question] why self.env.reset doesn't return an obs variable in FireResetEnv wrapper reset function, while it does in EpisodicLifeEnv and NoopResetEnv wrapper reset function
- Dominant language
- Python
- Stars
- 13.8k
- Forks
- 2.2k
- Avg merge
- 1h 35m
- Merged PRs (30d)
- 2
Description
### ❓ Question
In FireResetEnv wrapper in the file of stable-baselines3/stable_baselines3/common/atari_wrappers.py, I believe it is an issue that there is no obs return from self.env.reset(**kwargs)
The code is as in the below:
```
def reset(self, **kwargs) -> AtariResetReturn:
self.env.reset(**kwargs)
obs, _, terminated, truncated, _ = self.env.step(1)
if terminated or truncated:
self.env.reset(**kwargs)
obs, _, terminated, truncated, _ = self.env.step(2)
if terminated or truncated:
self.env.reset(**kwargs)
return obs, {}
```
I believe it should be modified to
```
def reset(self, **kwargs) -> AtariResetReturn:
self.env.reset(**kwargs)
obs, _, terminated, truncated, _ = self.env.step(1)
if terminated or truncated:
self.env.reset(**kwargs)
obs, _, terminated, truncated, _ = self.env.step(2)
if terminated or truncated:
obs, _ = self.env.reset(**kwargs)
return obs, {}
```
I have encountered an issue with GravitarNoFrameskip-v4 environment, when I run the command
`python train.py --algo dqn --env GravitarNoFrameskip-v4 --seed 37 --eval-freq -1 --n-timesteps 10000000 --tensorboard-log ./runs/`
on my computer, when it was running to the step 23374, when FireResetEnv reset execute
`obs, _, terminated, truncated, _ = self.env.step(2)`,
the environment terminated, and executed
```
if terminated or truncated:
self.env.reset(**kwargs)
```
the code will execute EpisodicLifeEnv reset() and NoopResetEnv reset(), however, **_the retruned obs variable from FireResetEnv reset is still obs before reset process_**,
`obs, _, terminated, truncated, _ = self.env.step(2)`
while I think it should be the obs after reset process.
```
if terminated or truncated:
obs, _ = self.env.reset(**kwargs)
```
Here is the datalog after I add some print lines:
```
EpisodicLifeEnv step(): lives: 4 -> 3, terminated: True, truncated: False
common/vec_env/dummy_vec_env.py: DummyVecEnv step_wait(): before env reset for env 0
FireResetEnv reset() called with kwargs: {'seed': None, 'options': None}
EpisodicLifeEnv reset() called with kwargs: {'seed': None, 'options': None}
EpisodicLifeEnv reset() end: lives: 3
EpisodicLifeEnv reset() end: obs shape: (210, 160, 3), obs.sum(): 3139748.00, obs.mean(): 31.148294
FireResetEnv step() called with action 1 after, terminated: False, truncated: False
FireResetEnv step() called with action 2 before, terminated: False, truncated: False
********** Monitor wrapper terminated: True, episode_num: 37, episode reward: 250.0, episode length: 2358, total steps: 96075 **********
EpisodicLifeEnv step(): lives: 3 -> 0, terminated: True, truncated: False
FireResetEnv step() called with action 2 after, terminated: True, truncated: False
EpisodicLifeEnv reset() called with kwargs: {'seed': None, 'options': None}
EpisodicLifeEnv performing full reset
NoopResetEnv reset() called with kwargs: {'seed': None, 'options': None}
********** Monitor reset() called
********** Monitor reset() with kwargs: {'seed': None, 'options': None} end
NoopResetEnv reset() end: performed 21 no-ops, obs shape: (210, 160, 3), obs.sum(): 211087
EpisodicLifeEnv reset() end: lives: 6
EpisodicLifeEnv reset() end: obs shape: (210, 160, 3), obs.sum(): 211087.00, obs.mean(): 2.094117
FireResetEnv reset() end
FireResetEnv reset() end: obs shape: (210, 160, 3), obs.sum(): 3122004.00, obs.mean(): 30.972262
```
### Checklist
- [x] I have checked that there is no similar [issue](https://github.com/DLR-RM/stable-baselines3/issues) in the repo
- [x] I have read the [documentation](https://stable-baselines3.readthedocs.io/en/master/)
- [x] If code there is, it is [minimal and working](https://github.com/DLR-RM/stable-baselines3/issues/982#issuecomment-1197044014)
- [x] If code there is, it is formatted using the [markdown code blocks](https://help.github.com/en/articles/creating-and-highlighting-code-blocks) for both code and stack traces.
Contributor guide
Assessment
This issue has not been assessed yet.