GPU VRAM Usage during training

Hi,

Thanks for your great work! I have some questions regarding the GPU usage when training with **LLaMa 2**:

1. What is the peak usage of the VRAM when training the Unlimiformer using the long-range training methods in both 8k and 16k settings?
2. Since the complexity is linear during training, training in 16k is around double VRAM than 8k if I understand correctly. So if I want to train Unilimiformer in 80k, it would be 10 times more VRAM usage than 8k?
3. I saw in a previous [issue](https://github.com/abertsch72/unlimiformer/issues/16) that currently Unilimiformer could only be trained in a single GPU, so the training length will be limited on the max single GPU RAM, say 80GB for A100 GPU. So I am curious is the 16k training length is the max possible length for now?

Thanks!