Multi GPU Memory keeps increasing while training TFT

• PyTorch-Forecasting version: 0.8.4
• PyTorch version: 1.8.1+cu102
• Python version: Python 3.6.11
• Operating System: Linux

Expected behavior

I follow the tft tutorial but want to train on multiple GPUs.

Actual behavior

RAM usage increase drastically over time until we get a memory error (Cannot allocate memory ...)

Changing to log_interval=-1 gets rid of the problem.
Also training on one GPU only doesn't increase RAM usage.

Code to reproduce the problem

Steps that differ from the tutorial:

1. Omit the "learning rate finder" part
2. add/replace these two lines in the pl.Trainer.
   gpus=[0, 1], accelerator='ddp',
3. Increase max_epochs and early stopping such that it doesn't stop early

/edit: For clarification: RAM usage keeps increasing, not VRAM (which is okay).

Use the TFT tutorial with PyTorch 0.8.4, PyTorch 1.8.1+cu102, and the reported multi-GPU Trainer settings. Compare DDP and single-GPU runs with log_interval enabled, watching host RAM rather than VRAM, then verify the log_interval=-1 workaround. Done means repeated multi-GPU training no longer grows RAM until allocation fails.