Slowness when using RNN

Hi,

I'm training agents on a custom environment (just one env, not vectorized) where each episode lasts a few hundred time steps, and each training iteration is done after 10 episodes are collected. Up until now, I was using MLP layers for both actor and critic models, and everything was fine. However, when I switched to an RNN layer, such as LSTM, there was an extreme slowdown, the figure below for reference:

I know that RNNs are slower, but this seems a bit too much of a slowdown, especially in a simple environment. I suppose that, in part, this is because the LSTM takes as input the whole sequence. In the [configuration file of the LSTM](https://github.com/facebookresearch/BenchMARL/blob/main/benchmarl/conf/model/layers/lstm.yaml) is not possible to customize the length of the sequence.
Do you have any suggestions on how to speed up the learning in the case of RNN?

Start with benchmarl/conf/model/layers/lstm.yaml and reproduce the reported comparison using the single custom environment, few-hundred-step episodes, and ten episodes per iteration. Trace how the LSTM receives the full sequence and measure where training slows relative to the MLP; done means an agreed, tested way to control or avoid the bottleneck.

Written by the indexing model from the issue text.