memory growth during training

Ram is growing during train loop
I'm training arcface in recognition/arcface_torch.

`torchrun --nproc_per_node 2 train_v2.py configs/custom_ds_pfc03_40epoch_2gpu_vits` - script launch

```
from easydict import EasyDict as edict

config = edict()
config.margin_list = (1.0, 0.0, 0.4)
config.network = "vit_s_dp005_mask_0"
config.resume = False
config.output = None
config.embedding_size = 128
config.sample_rate = 0.3
config.fp16 = False
config.weight_decay = 0.1
config.batch_size = 64
config.optimizer = "adamw"
config.lr = 0.001

config.verbose = 2000
config.dali = False

config.rec = "/var/ssd/face/"
config.num_classes = 1_000_000
config.num_image = 5_000_000
config.num_epoch = 20
config.warmup_epoch = config.num_epoch // 10
config.val_targets = []
```
my config.

Script takes some memory on load (~20 gb) and then progressively grows to full (48gb) and script fails.

I tried to lower number of workers, prefetch size (didn't work) and ram still grows.