VideoReader initialization becomes very slow after loading many videos

Decord installed with pip, observed using`VideoReader` alongside the Pytorch dataloader for decoding videos.
Code looks something like:

```
class Dataset
def __getitem__(self, idx):
vr = VideoLoader(self.video_paths[idx], num_threads=1, device=cpu(0))
images = vr.get_batch([...]).asnumpy()
return images
```
However, after some number of examples, (dependent on num_workers), creating the VideoLoader takes up to 10 seconds, my CPU utilization drops significantly and dataloading becomes 3-4x slower.

I've tried including `del vr` and `gc.collect()` as from https://github.com/dmlc/decord/issues/1#issuecomment-1437374584. I've also attempted to set the environment variable `DECORD_NUM_THREADS` as well different `num_workers` in the pytorch dataloader.