DLR-RM / DLR-RM/stable-baselines3

BetaDistribution policy for bounded continuous action spaces to avoid Gaussian clipping bias and improve training stability

Open
#2,142 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
13.8k
Forks
2.2k
Avg merge
1h 35m
Merged PRs (30d)
2

Description

### 🚀 Feature

Add a option of `BetaDistribution` policy for bounded continuous action spaces to avoid Gaussian clipping bias and improve training stability.

### Motivation

[ Petrazzini & Antonelo (2021) ](https://arxiv.org/pdf/2111.02202) demonstrated that replacing Gaussian with a Beta distribution (compact support) in PPO yields significantly faster convergence, higher final rewards, and a 63 % increase in success on CarRacing-v0.

[Improving Stochastic Policy Gradients in Continuous Control with Deep
Reinforcement Learning using the Beta Distribution](https://proceedings.mlr.press/v70/chou17a/chou17a.pdf) showed that the Beta policy is bias-free and provides significantly faster convergence and higher scores over the Gaussian policy when both are used with trust region policy optimization (TRPO) and actor critic with experience replay (ACER).

### Pitch

New `BetaDistribution` class in `stable_baselines3.common.distributions`, mirroring `DiagGaussianDistribution` with methods and optionally to use it on any `on policy algorithm`.

Or create combination of Gaussian and beta, where for bounded actions, it will use Beta distribution and for unbounded the Gaussian distribution.

### Alternatives

- **Tanh-squashed Gaussian (SquashedDiagGaussianDistribution):** still biases density near ±1 and can have vanishing gradients at the tails .
- **Truncated normal**: more complex to implement and less stable under backpropagation .
- **State-Dependent Noise (gSDE)**: adapts variance but still uses Gaussian support, so does not fully eliminate bias at hard bounds .

### Additional context

_No response_

### Checklist

- [x] I have checked that there is no similar [issue](https://github.com/DLR-RM/stable-baselines3/issues) in the repo
- [x] If I'm requesting a new feature, I have proposed alternatives

@araffin What would be the best implementation?

Contributor guide

Open the contributing guide

Research direction

Start in stable_baselines3.common.distributions by reading DiagGaussianDistribution and the related on-policy algorithm integration points. Before implementation, clarify whether the project wants a standalone BetaDistribution or a Gaussian/Beta combination; done should include a defined way to use the policy for bounded continuous actions and evidence that the intended training-stability behavior is supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.