[Feature Request] Ability to skip transitions based on masks

Motivation

Certain simulators (such as Isaac Gym and Isaac Sim) create several environment instances as part of the same stage. Unlike CPU-based vectorization, where each environment instance, has its own stepping, in GPU-based parallelized environments, the stepping is common. This leads to an issue when certain environments want to have dummy steps for the scene to come to rest. For instance, we often spawn deformable bodies (such as cloths) from a height and let them fall onto the ground and come to rest before we start learning.

There isn't a clear mechanism to have these dummy steps in the simulators and probably this functionality won't be added since it ties deeply into the whole parallelization philosophy (i.e. all environments are just a huge single GPU data). Thus, this issue would be best handled by the learning framework itself.

Solution

A possible way to deal with this issue could be having the ability to add masking of transition tuples. For instance, if out of N environments, (2, 5, 10) indices were last reset, we enable their masking (additional quantity to "infos" dictionary) for certain time steps. If the masks evaluate to true, then the trajectory or replay buffer doesn't add transitions for those indices into the memory.

Alternatives

I couldn't really think of any other alternatives here but would be interested to hear your thoughts on this.

Additional context

• Proposed in Orbit #64

Checklist

• I have checked that there is no similar issue in the repo (required)