Right-size and expose model-proxy connection pools

Parent workstream

#3002 — Performance workstream: model servers

Slowdown path

flowchart LR
    A[Model-proxy worker receives requests] --> B[Shared per-process connector is saturated]
    B --> C[Calls wait before reaching an engine host]
    C --> D[Engine scheduler cannot observe the queued demand]

Current behavior

The shared aiohttp connector has a total limit and a per-host limit. Both configured aggregate values are divided by FastAPI worker count when each process creates its client (server_utils.py lines 95–112 and server_utils.py lines 155–180).

VLLMModel creates one NeMoGymAsyncOpenAI wrapper per configured engine URL (vllm_model/app.py lines 290–310). Sessions are deterministically mapped to one of those clients, but every wrapper ultimately sends through Gym's single process-global aiohttp client (vllm_model/app.py lines 1500–1512 and openai_utils.py lines 1043–1084). Therefore the total connector limit can bind across engine hosts before any one per-host limit binds.

Add model-proxy-specific capacity diagnostics that combine worker count, engine destination count, intended proxy concurrency, total connector limit, and per-host limit. Export connector queue time through the shared instrumentation in #3008 and make effective model-proxy capacity visible at startup.

Impact and provenance

Measured

The investigated Lightning configuration used an aggregate total limit of 4,096 across 16 workers, which arithmetically yields 256 sockets per worker. The workload estimate was about 512 concurrent engine calls per worker, suggesting a possible 2:1 client-side queue.

Expected

That 2:1 queue is a hypothesis, not a measured result. No connector-acquisition telemetry currently proves that all estimated calls overlap, that the total limit is continuously saturated, or that raising it improves throughput. The expected outcome of this issue is first to verify or falsify the hypothesis. A throughput gain is expected only if measured connector wait is material and engine capacity remains available.

Implementation constraints

• Preserve deterministic per-session engine selection and endpoint-rebinding behavior.
• Size both the total connector and per-host limits. Multiple engine hosts do not avoid a too-small total limit.
• Respect file-descriptor, ephemeral-port, backend-protection, and deployment-side connection budgets.
• Keep configured values aggregate across workers unless a deliberate compatibility change is approved; always show effective per-process values.
• Do not make model-proxy queues disappear by moving uncontrolled load into engine schedulers.
• Reuse #3008's low-overhead shared instrumentation rather than creating a second incompatible queue metric.

Acceptance criteria

• Startup reports configured aggregate limits, effective per-worker limits, engine destination count, and the sizing assumptions used.
• A configuration whose intended per-worker demand exceeds total or per-host capacity produces an actionable diagnostic.
• Connector queue duration is visible for proxy-to-engine calls and separable from engine service time.
• The Lightning 4,096/16-worker scenario is reproduced or falsified with observed concurrent calls and queue metrics.
• Tests cover one and many engine URLs, 1 and many workers, uneven destination load, and integer-division edge cases.
• Session affinity and dynamic endpoint rebinding remain correct.
• Documentation explains descriptor and ephemeral-port tradeoffs rather than recommending an unbounded limit.

Benchmark plan

1. Run a proxy against 1, 4, and 16 engine destinations with controlled service latency and concurrency.
2. Test the investigated 4,096 aggregate limit with 16 proxy workers, then limits below and above observed peak in-flight engine calls.
3. Record actual concurrent proxy-to-engine calls, connector queue p50/p95/p99, engine scheduler queue/service time, proxy throughput, end-to-end latency, open sockets, file descriptors, and errors.
4. Repeat with skewed session affinity so one host reaches limit_per_host while total capacity remains.
5. Report whether the proposed 2:1 queue existed and whether any limit increase moved the bottleneck or improved throughput.

Related work

• #3008 provides shared connector-queue telemetry and generic capacity validation.
• #3014 may change upstream agent fan-out and therefore proxy demand.
• #3002 tracks model-server performance work.

Start with nemo_gym/server_utils.py, nemo_gym/openai_utils.py, and responses_api_models/vllm_model/app.py at the linked sections, then review #3008's shared instrumentation. Reproduce the one- and many-destination benchmark scenarios with varied worker counts and limits, measuring connector queue and engine timings. Done means actionable startup capacity diagnostics, queue metrics, coverage for the listed affinity and division cases, preserved rebinding behavior, and documentation of connection-budget tradeoffs.