OOM with bf16-true, Quantization, for long context length.

Nobody has claimed this yet.

Hello.

I tried to install everything mentioned from git cloning to installing all requirements, except flash attentions.

I have an A100 40GB machine
CUDA 11.0

Now downloaded the llama-2-7B checkpoints and prepared.

also prepared alpaca dataset. with llama-2-7B using
```
python scripts/prepare_alpaca.py
```

Now i changes the finetune.lora.py. Inside get_batch.
```
max_len = 4000 # max(len(s) for s in input_ids) if fabric.device.type != "xla" else longest_seq_length
```

Using the Below command runs successfully on single GPU. with 4000 context length
```
python finetune/lora.py --precision 'bf16-true'
```

Using Below command with quantization is giving OOM.
```
python finetune/lora.py --precision 'bf16-true' --quantize 'bnb.nf4-dq'
```

This is suspicious. I think if just "bf16-true" is running fine, then "bf16-true" along with quantization should also work even with less memory.

Can someone suggest, what went wrong. What am i missing?

Start with finetune/lora.py, especially get_batch, and reproduce both commands after preparing the Alpaca data with scripts/prepare_alpaca.py. Compare the memory behavior of bf16-true with and without bnb.nf4-dq; done means identifying and documenting the cause of the quantized run's OOM or establishing a specific change needed to avoid it.

Written by the indexing model from the issue text.