Strange CUDA: OOM (Out of Memory) Issue

Hi, So I'm currently trying to train SAM with different configurations.
So I've modified the depth and patch size in the codes, I'm facing one strange issue.
Below is the vit-h (ignore attention indexes...)
and Below is the vit-b:

This is the patch size (kept same for both)

To my surprise, the model with vit-h is training fine:
But the model with vit-b keeps giving me OOM Issue:

I'm confused, how and why! Vit-B is lighter than Vit-H, if Vit-H is getting trained, then Vit-B should train as well.

BTW, I'm also training the Image encoder in both cases.

No source file, test, command, or complete configuration is named; start by reproducing the reported CUDA OOM with the modified ViT-B and ViT-H depth and patch-size settings. Collect the exact model, batch, image, GPU, and memory details, then determine whether the discrepancy is reproducible and document a confirmed cause or minimal reproduction.

Written by the indexing model from the issue text.