Concurrent TTS Performance Observation (VoxCPM2 + vLLM-Omni)

Hi Team,

I tested the /api/tts_direct_stream endpoint under concurrent load and observed the following behavior.

Environment:

Server: AWS EC2 (L40S GPU)
Serving Stack: vLLM-Omni v0.20.0
Endpoint: /api/tts_direct_stream
Client: Python (requests + ThreadPoolExecutor) for concurrent load testing

Model Serving Command:

vllm-omni serve $model_path
--omni
--served-model-name tts-service
--host 0.0.0.0
--port 8000
--enforce-eager
--dtype bfloat16

Test Results:

1 concurrent request
Latency: ~0.59s
TTFB: ~0.11s

10 concurrent requests
Average latency: ~3.08s
Maximum latency: ~4.46s
TTFB increases from ~0.49s for the first few requests to ~3.53s for the last requests.

It appears that requests are being processed in batches rather than all 10 starting simultaneously. The first few requests begin processing immediately, while the remaining requests wait before inference starts, resulting in higher TTFB and overall latency.

Could you please confirm:

-Is there a concurrency limit (or maximum number of parallel inference slots) configured in the current vLLM-Omni/VoxCPM2 setup?
-Is this expected behavior for the current deployment, or is there any server-side configuration that can improve concurrent throughput?
-What's the minimum GPU for getting concurrency.
-Are there recommended settings (e.g., scheduler, batching, max_num_seqs, or deployment configuration) to handle higher concurrent TTS requests?

Thanks.

Start by reproducing concurrent requests against /api/tts_direct_stream using the Python requests and ThreadPoolExecutor setup and the listed vLLM-Omni serve command. Compare latency and TTFB across concurrency levels, then document whether the behavior is expected, what concurrency limits apply, and which supported settings or minimum GPU guidance improve throughput.