DLR-RM / DLR-RM/stable-baselines3
[Question] Why does VecNormalize normalize rewards by returns instead of rewards?
- Dominant language
- Python
- Stars
- 13.8k
- Forks
- 2.2k
- Avg merge
- 1h 35m
- Merged PRs (30d)
- 2
Description
### ❓ Question
I just read the code of `VecNormalize` and was surprised by the way the reward normalization is computed. I had assumed the wrapper to keep track of past rewards, storing their running mean and variance. New rewards would then be normalized using this statistic. Instead the wrapper appears to normalize the rewards using statistics of the *returns*, i.e. the discounted sums over rewards. This is done in the `_update_reward` function. For environments with only positive rewards (like CartPole) the value of the return increases as the episode progresses, particularly for large values of gamma, while the value of the reward does not depend on the time step within the episode. Normalizing rewards by statistics over returns seems counterintuitive to me, so I wanted to ask if this is a bug or intended behaviour
### Checklist
- [x] I have checked that there is no similar [issue](https://github.com/DLR-RM/stable-baselines3/issues) in the repo
- [x] I have read the [documentation](https://stable-baselines3.readthedocs.io/en/master/)
- [x] If code there is, it is [minimal and working](https://github.com/DLR-RM/stable-baselines3/issues/982#issuecomment-1197044014)
- [x] If code there is, it is formatted using the [markdown code blocks](https://help.github.com/en/articles/creating-and-highlighting-code-blocks) for both code and stack traces.
Contributor guide
Research direction
Start by reading the VecNormalize implementation, focusing on the _update_reward function and how its return statistics are used to normalize rewards. Compare that behavior with the documented expectations and the CartPole example described in the issue. Done means determining whether return-based normalization is intentional and documenting or correcting the behavior accordingly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100