Stuck when training with DDPG

❓ Question

Hello,
When I use DDPG to train the gym third party environment torcs (a racing environment), after a period of time when the program starts to run, it stalls for a while and then the car keeps turning to the left, no problem when using stable_baselines, this happens when using stable_baselines3

TIMESTEPS = int(1e7) + 1

if __name__ == "__main__":
    env = gym.make('Torcs-v0')
    n_actions = env.action_space.shape[-1]
    action_noise = NormalActionNoise(mean=np.zeros(n_actions),sigma=0.1 * np.ones(n_actions))

    model = DDPG(policy="MlpPolicy",
                 env=env,
                 action_noise=action_noise,
                 verbose=1,
                 tensorboard_log="./ddpg_tensorboard/"
                 )
    model.learn(
        total_timesteps=TIMESTEPS,
    )

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.

Start by reproducing the provided DDPG example with the Gym Torcs-v0 environment and compare its behavior with stable_baselines. Investigate whether the stall and persistent left turn can be reproduced in stable-baselines3, then document a confirmed cause or workaround; the issue does not name a failing test or source file.