Memory leak during training

1. Full runnable code or full changes you made:

I am using Regionclip repo which is based detectron2, and I found that the whole training process exists memory leak.
I am wondering how to locate the root of the memory leak, by the way, I am using default distributed training on 8 gpu and use torch.multiprocessing.set_sharing_strategy('file_system') in sharing_strategy

**I am wondering which maybe the possible cause of mem leak, can anyone give me some clue?**

3. What exact command you run:

I am using regionclip ovod training script and set the eval period = 5000, i log the free cpu mem and plot it below.

> python3 ./tools/train_net.py \
> --num-gpus 1 \
> --config-file ./configs/COCO-InstanceSegmentation/CLIP_fast_rcnn_R_50_C4_ovd.yaml \
> MODEL.WEIGHTS ./pretrained_ckpt/regionclip/regionclip_pretrained-cc_rn50.pth \
> MODEL.CLIP.OFFLINE_RPN_CONFIG ./configs/COCO-InstanceSegmentation/mask_rcnn_R_50_C4_1x_ovd_FSD.yaml \
> MODEL.CLIP.BB_RPN_WEIGHTS ./pretrained_ckpt/rpn/rpn_coco_48.pth \
> MODEL.CLIP.TEXT_EMB_PATH ./pretrained_ckpt/concept_emb/coco_48_base_cls_emb.pth \
> MODEL.CLIP.OPENSET_TEST_TEXT_EMB_PATH ./pretrained_ckpt/concept_emb/coco_65_cls_emb.pth \

4. __Full logs__ or other relevant observations:

> Traceback (most recent call last):
> File "/home/wulei04/miniconda3/envs/detic/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
> obj = _ForkingPickler.dumps(obj)
> File "/home/wulei04/miniconda3/envs/detic/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
> cls(buf, protocol).dump(obj)
> File "/home/wulei04/miniconda3/envs/detic/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 312, in reduce_storage
> metadata = storage._share_filename_()
> RuntimeError: error executing torch_shm_manager at "/home/wulei04/miniconda3/envs/detic/lib/python3.8/site-packages/torch/bin/torch_shm_manager" at ../torch/lib/libshm/core.cpp:99
> Traceback (most recent call last):
> File "/home/wulei04/miniconda3/envs/detic/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
> obj = _ForkingPickler.dumps(obj)
> File "/home/wulei04/miniconda3/envs/detic/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
> cls(buf, protocol).dump(obj)
> File "/home/wulei04/miniconda3/envs/detic/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 312, in reduce_storage
> metadata = storage._share_filename_()
> RuntimeError: error executing torch_shm_manager at "/home/wulei04/miniconda3/envs/detic/lib/python3.8/site-packages/torch/bin/torch_shm_manager" at ../torch/lib/libshm/core.cpp:99
> Traceback (most recent call last):
> File "/home/wulei04/miniconda3/envs/detic/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
> obj = _ForkingPickler.dumps(obj)
> File "/home/wulei04/miniconda3/envs/detic/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
> cls(buf, protocol).dump(obj)
> File "/home/wulei04/miniconda3/envs/detic/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 312, in reduce_storage
> metadata = storage._share_filename_()
> RuntimeError: error executing torch_shm_manager at "/home/wulei04/miniconda3/envs/detic/lib/python3.8/site-packages/torch/bin/torch_shm_manager" at ../torch/lib/libshm/core.cpp:9
>

## Environment:

GPU 0,1,2,3,4,5,6,7 GeForce RTX 2080 Ti (arch=7.5)
CUDA_HOME None - invalid!
Pillow 9.2.0
torchvision 0.9.2+cu111 @/home/wulei04/miniconda3/envs/detic/lib/python3.8/site-packages/torchvision
torchvision arch flags /home/wulei04/miniconda3/envs/detic/lib/python3.8/site-packages/torchvision/_C.so
fvcore 0.1.5.post20220512
iopath 0.1.8
cv2 4.6.0
---------------------- ----------------------------------------------------------------------------------------
PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
- CuDNN 8.0.5
- Magma 2.5.2