Sanity check: VRAM usage on llama-2-7b-chat-hf higher than without Unlimiformer on low tokens?

I'm trying out the new Unlimiformer llama-2 code on llama-2-7b-chat-hf, on a 24GB 3090.
I understand Unlimiformer probably wasn't created with consumer GPUs in mind, but I'd hoped I'd be able to squeeze some more context out of my GPU locally before having to resort to expensive cloud GPUs.
I managed to get everything working, but the VRAM usage per token seems to be higher than on stock llama-2-7b-hf.
I imagine there is expected overhead from running Unlimiformer, though it is more than I expected.
With vanilla Transformers (same versions and everything) on fp16, I can ingest up to ~5350 tokens at once before running out of memory.
With Unlimiformer, 5350 tokens runs out of memory, and I can barely do more than 4096 tokens (5000 OOMs).
Is this expected overhead? And is this overhead fixed, or does it vary with the model size?

Semi-related side-questions: Is there anything Unlimiformer does that would prevent it from working with bitsandbytes 8/4 bit quanitzation, or should that be a matter of simply enabling it? And should training QLoRA with peft work?