Issues with VLLM Integration Speedup

Nobody has claimed this yet.

Hello,

I've been trying to work with the [[vLLM integration](https://github.com/lm-sys/FastChat/blob/main/docs/vllm_integration.md)] and I'm facing some performance discrepancies. According to the documentation, I should achieve a significant speedup, but in my tests, I'm seeing different results:

Directly running with FastChat: 16 t/s
Using the VLLM integration: 25 t/s (only 1.5x speedup)
VLLM offline inference: 90 t/s (expected 6x speedup)
I'm running Vicuna33b on a gin H100 gpu. Has anyone experienced this before? Are there any additional configurations or tweaks I might be missing to get the desired speedup?

Thanks in advance for any guidance or advice!

Start with docs/vllm_integration.md and compare its documented setup with the reported Vicuna-33B benchmarks on an H100. Reproduce the three inference paths and inspect the configuration differences; done means identifying the cause of the speed discrepancy or documenting the required settings and expected results.

Written by the indexing model from the issue text.