Running multiple instances on a single GPU occationally get empty frame

Hi,

I am trying to run multiple AI2THOR instances on a single NVIDIA A6000 GPU. I have correctly configured the x server, and successfully tested my code under the single process setting. When I turn to launching multiple instances, I found that although the controllers step forward without reporting any bug, the returned frame occationally becomes empty (every pixel becomes [35, 31, 32] which I assume is the background color). I have ensured that the resources are enough for running the instances. I also tried using the nvidia MPS service to isolate the processes but the bug still exists. Any idea or suggestions on how to fix this?

BTW I am using and have to use AI2THOR 3.1.0.

Thank you very much