Multiple GPUs memory issue

I'm trying llama-cpp-python (v0.2.18) with Nvidia GPUs. I have 4 T4 GPUs to load a large size model which does not fit in one T4. Looks like right after llm = llama_cpp.Llama(...), the memory of the 4 GPUs are evenly distributed. However when I do the prompt and output generation using llm(prompt, max_tokens=1000, ...), the main GPU (device 0) gets more than doubled memory usage and then out-of-memory happens. This is not happening with the llamma.cpp command-line using "./main --n-gpu-layers 15000 -m dolphin-2_2-yi-34b.Q5_K_M.gguf ....". Can anyone help me solve this? I'd prefer to keeping using the python wrapper since it's more convenient than starting a server endpoint. Thanks!