DLR-RM / DLR-RM/stable-baselines3

[Question] Why does VecNormalize normalize rewards by returns instead of rewards?

Open
#2,279 0 comments 0 reactions 0 assignees View on GitHub
duplicate question
Dominant language
Python
Stars
13.8k
Forks
2.2k
Avg merge
1h 35m
Merged PRs (30d)
2

Description

### ❓ Question

I just read the code of `VecNormalize` and was surprised by the way the reward normalization is computed. I had assumed the wrapper to keep track of past rewards, storing their running mean and variance. New rewards would then be normalized using this statistic. Instead the wrapper appears to normalize the rewards using statistics of the *returns*, i.e. the discounted sums over rewards. This is done in the `_update_reward` function. For environments with only positive rewards (like CartPole) the value of the return increases as the episode progresses, particularly for large values of gamma, while the value of the reward does not depend on the time step within the episode. Normalizing rewards by statistics over returns seems counterintuitive to me, so I wanted to ask if this is a bug or intended behaviour

### Checklist

- [x] I have checked that there is no similar [issue](https://github.com/DLR-RM/stable-baselines3/issues) in the repo
- [x] I have read the [documentation](https://stable-baselines3.readthedocs.io/en/master/)
- [x] If code there is, it is [minimal and working](https://github.com/DLR-RM/stable-baselines3/issues/982#issuecomment-1197044014)
- [x] If code there is, it is formatted using the [markdown code blocks](https://help.github.com/en/articles/creating-and-highlighting-code-blocks) for both code and stack traces.

Contributor guide

Open the contributing guide

Research direction

Start by reading the VecNormalize implementation, focusing on the _update_reward function and how its return statistics are used to normalize rewards. Compare that behavior with the documented expectations and the CartPole example described in the issue. Done means determining whether return-based normalization is intentional and documenting or correcting the behavior accordingly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.