Memory increase

I used Ctranslate2-quantized version of fastchat-t5 (https://huggingface.co/limcheekin/fastchat-t5-3b-ct2), as the LLM of a question answering system. The QA system is wrapped in Rest API. The model works really well. But an issue I notice is that the GPU memory footprint increases over time (requests), and eventually causing OOM error.

In my case, I set max_batch_size=1, max_input_size=2048, max_decoding_size=1024. The GPU is L4 with 24 GB ram (should be more than enough for the model which only takes 3GB once loaded).

I was thinking about the following solutions. Can you please provide some suggestions?

1. Separate the LLM from the QA system, spin up a Rest API just for the LLM, and call the API endpoint in the QA system. This is because I notice many hosting services like vLLM claims better throughput handling. Will CTranslate2 model benefit from doing so?

2. I notice in the OpenNMT-py RestAPI server, the model is unloaded to cpu and reload, based on a timer. When I try it out, the unload and load take a couple of seconds. Doing this every a few requests, does not seem to be efficient.

Thank you.

No repository files or tests are named. Start by reproducing repeated requests with the stated CTranslate2-quantized model and limits (max_batch_size=1, max_input_size=2048, max_decoding_size=1024), then inspect GPU memory behavior in the Rest API setup. Done means isolating whether memory growth is reproducible and identifying a repository-specific cause or a documented limitation.