Out of Memory

I'm able to run this model as cpu only model. I was expecting to do a split between gpu/cpu ram for the model under gguf, but regardless of what -n

or even if I input

(textgen) [root@pve0 bin]# ./main
Log start
main: build = 1233 (98311c4)
main: seed = 1694805687

CUDA error 2 at /data/llama.cpp/ggml-cuda.cu:5509: out of memory
current device: 0
(textgen) [root@pve0 bin]#

1041 cmake .. -DLLAMA_CUBLAS=ON
1042 cmake --build . --config Release
1043 cd ..
1044 history
1045 ./main -m /data/text-generation-webui/models/Marx-3B-V2-Q4_1-GGUF.gguf -n 128
1046 cd build/
1047 cd bin/
1048 ./main -m /data/text-generation-webui/models/Marx-3B-V2-Q4_1-GGUF.gguf -n 128
1049 ./main -m /data/text-generation-webui/models/Marx-3B-V2-Q4_1-GGUF.gguf -n 32
1050 ./main -m /data/text-generation-webui/models/Marx-3B-V2-Q4_1-GGUF.gguf -n 8
1051 ./main -m /data/text-generation-webui/models/Marx-3B-V2-Q4_1-GGUF.gguf -n 0

(textgen) [root@pve0 bin]# ./main -m /data/text-generation-webui/models/Marx-3B-V2-Q4_1-GGUF.gguf -n 0
Log start
main: build = 1233 (98311c4)
main: seed = 1694804999

CUDA error 2 at /data/llama.cpp/ggml-cuda.cu:5509: out of memory
current device: 0
(textgen) [root@pve0 bin]# ./main -m /data/text-generation-webui/models/Marx-3B-V2-Q4_1-GGUF.gguf -n 0
(textgen) [root@pve0 bin]# nvidia-smi
Fri Sep 15 12:11:15 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.43.04 Driver Version: 515.43.04 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P1000 Off | 00000000:05:00.0 Off | N/A |
| 37% 50C P8 N/A / N/A | 0MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
(textgen) [root@pve0 bin]# ls mode
(textgen) [root@pve0 bin]# ./main -m /data/text-generation-webui/models/Llama-2-7b-chat-hf -n 0
Llama-2-7b-chat-hf/ Llama-2-7b-chat-hf-ct/
(textgen) [root@pve0 bin]# ./main -m /data/text-generation-webui/models/Llama-2-7b-chat-hf -n 0
Llama-2-7b-chat-hf/ Llama-2-7b-chat-hf-ct/
(textgen) [root@pve0 bin]# ./main -m /data/text-generation-webui/models/llama-2-7b-chat.ggmlv3.q2_K.bin -n 0
Log start
main: build = 1233 (98311c4)
main: seed = 1694805567

CUDA error 2 at /data/llama.cpp/ggml-cuda.cu:5509: out of memory
current device: 0

Oracle linux 8.5
python 3.10
torch 2.0.1
cuda 11.7
nvidia-smi shows cuda 11.6 compatible