Unable to run AI2Thor in parallel on multiple GPUs on a node

I have some fixed trajectories. When I run ai2thor (version 2.1.0) inside a docker container and try to replicate some trajectories, it happens that sometimes the length of the metadata['colorBounds'] goes as huge as 20,000 and the container occupies huge amount of memory and the processing becomes extremely slow. The same image at other instances runs fine for some trajectories and then gets stuck. Yet at other instances the same image runs fine for all trajectories.

I have attached the logs (though I could not spot anything unusual in the logs).
[AI2-Thor.zip](https://github.com/allenai/ai2thor/files/5251017/AI2-Thor.zip)

The steps I follow are:
1. Create a docker image and run the container.
2. Inside the container, I start X server on a particular GPU.
3. Run the code on the same GPU.

(CUDA_VERSION=10.0). Here is the [Dockerfile](https://github.com/allenai/ai2thor/files/5251026/Dockerfile.txt)