Memory Problem

I wrote a small script to extract features from my spesific dataset. My intention was to use those features in Anomaly detection task. I realize a small problem. I extract 64 frames with 240,320 spatial dimensions. When I feed network with this input for test purpose, memory usage is nearly 1GB(for Kinetics/SLOWFAST_4x16_R50.yaml). When I scale my input to have 224 in small dimension, memory usage increase to 10GB. I thought there is a problem with your scale function but when I tried different sizes such that 236,256, I saw memory usage again 1GB. Is there a specific reason for this increase in the memory when size is fixed to 224? I used Tesla P100 during inference.

Additionally, when I try a second run in size=224 case, memory usage for second run is also 1GB.