DLR-RM / DLR-RM/stable-baselines3
BetaDistribution policy for bounded continuous action spaces to avoid Gaussian clipping bias and improve training stability
- Dominant language
- Python
- Stars
- 13.8k
- Forks
- 2.2k
- Avg merge
- 1h 35m
- Merged PRs (30d)
- 2
Description
### 🚀 Feature
Add a option of `BetaDistribution` policy for bounded continuous action spaces to avoid Gaussian clipping bias and improve training stability.
### Motivation
[ Petrazzini & Antonelo (2021) ](https://arxiv.org/pdf/2111.02202) demonstrated that replacing Gaussian with a Beta distribution (compact support) in PPO yields significantly faster convergence, higher final rewards, and a 63 % increase in success on CarRacing-v0.
[Improving Stochastic Policy Gradients in Continuous Control with Deep
Reinforcement Learning using the Beta Distribution](https://proceedings.mlr.press/v70/chou17a/chou17a.pdf) showed that the Beta policy is bias-free and provides significantly faster convergence and higher scores over the Gaussian policy when both are used with trust region policy optimization (TRPO) and actor critic with experience replay (ACER).
### Pitch
New `BetaDistribution` class in `stable_baselines3.common.distributions`, mirroring `DiagGaussianDistribution` with methods and optionally to use it on any `on policy algorithm`.
Or create combination of Gaussian and beta, where for bounded actions, it will use Beta distribution and for unbounded the Gaussian distribution.
### Alternatives
- **Tanh-squashed Gaussian (SquashedDiagGaussianDistribution):** still biases density near ±1 and can have vanishing gradients at the tails .
- **Truncated normal**: more complex to implement and less stable under backpropagation .
- **State-Dependent Noise (gSDE)**: adapts variance but still uses Gaussian support, so does not fully eliminate bias at hard bounds .
### Additional context
_No response_
### Checklist
- [x] I have checked that there is no similar [issue](https://github.com/DLR-RM/stable-baselines3/issues) in the repo
- [x] If I'm requesting a new feature, I have proposed alternatives
@araffin What would be the best implementation?
Contributor guide
Research direction
Start in stable_baselines3.common.distributions by reading DiagGaussianDistribution and the related on-policy algorithm integration points. Before implementation, clarify whether the project wants a standalone BetaDistribution or a Gaussian/Beta combination; done should include a defined way to use the policy for bounded continuous actions and evidence that the intended training-stability behavior is supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100