pretrain custom dataset gpu memory oom

Nobody has claimed this yet.

I use an A100 40G to pretrain my own data, but the gpu memory oom will appear. Below is my data and configuration file. In particular, my batch size has been set to 1.
[data and configuration.zip](https://github.com/Lightning-AI/litgpt/files/15419174/data.and.configuration.zip)

CUDA_VISIBLE_DEVICES=0 litgpt pretrain --model_name Llama-2-7b-hf --data CustomData --config ../config_hub/pretrain/llama.yaml

The strange thing is that I used the data([SlimPajama] on https://github.com/Lightning-AI/litgpt/blob/main/tutorials/pretrain_tinyllama.md) provided by the repo to train on 4x4090 24G and no GPU memory oom appeared. The following is the startup command I used.
CUDA_VISIBLE_DEVICES=0,1,2,3 litgpt pretrain --model_name Llama-2-7b-hf --config ./config_hub/llama.yaml

Is there any difference in the GPU memory requirements between single GPU and multi-GPU pretrain?

Start by unpacking data and configuration.zip and comparing config_hub/pretrain/llama.yaml with config_hub/llama.yaml. Reproduce the single-GPU and four-GPU commands, then determine and document what accounts for the differing GPU memory requirements and whether the custom dataset or configuration triggers the OOM.

Written by the indexing model from the issue text.