Memory Concerns with Vicuna-1.5-13B-16k Model via `openai` API

Nobody has claimed this yet.

Hello FastChat / LMSYS Team,

I want to express our gratitude for the tools and models you've developed, which have been incredibly useful for our tasks.
The usability is great and everything has been functioning as expected.

However, we've hit a roadblock recently.
After using the Vicuna-1.5-13B-16k model (hosted locally via the FastChat API) through the `openai` package for a while, we're encountering memory issues.
Specifically, we're running into out of memory problems over extended periods of usage.

We've tried a couple of things to address this:
- We attempted to distribute the model over two A100s (40GB each), but unfortunately, that didn't provide a solution. Although we haven't gone beyond two instances, we believe that two A100 GPUs with 40GB each should be more than sufficient for inference.
- We also experimented with the `use_cache` parameter in the model's `config.json`, but it didn't yield significant improvements.

We're reaching out to see if you might have any insights or suggestions on how to tackle this ongoing memory issue.

Thank you for your time and assistance.

Best regards,
Patrick

Start by reproducing the long-running Vicuna-1.5-13B-16k workload through the FastChat API using the `openai` package, while checking the `use_cache` setting in `config.json`. Done means identifying a confirmed source of the extended-use out-of-memory behavior or documenting a reproducible, actionable diagnosis.

Written by the indexing model from the issue text.