Connection pool: treat EADDRNOTAVAIL as backpressure so the full-budget auto ceiling is safe under max_throughput bursts

Follow-up to #411 (pacing + bounded auto warmup), based on empirical validation reported in https://github.com/mlcommons/endpoints/pull/411#issuecomment-4983736498.

Context

#411's connect pacing removes the t=0 SYN flood (server-side half of the problem), but the auto max_connections ceiling is 100% of the ephemeral-port budget. In offline max_throughput, pool-growth demand is the whole dataset burst, so the pool grows to its ceiling — and a ceiling equal to the full range makes the last connect wave race the host's remaining sockets/TIME_WAIT. On a co-located single-endpoint setup this drops a small percentage of samples as EADDRNOTAVAIL in a ~1–2 s burst at t=0 (down from majority-failing pre-#411; an explicit max_connections=16384 reaches zero failures with no throughput regression).

A failed connect() currently propagates and fails the query — the port race costs samples instead of latency.

Chosen direction: dynamic backpressure, not a static cap

A static auto-ceiling fraction (as #407 proposed, e.g. 0.5 × budget) was considered and rejected: it caps the pool for the entire run duration even when the port space is free. The transient t=0 contention should instead be absorbed dynamically — the kernel is the only correct accountant of port availability, and its answer changes over the run.

Fix: port exhaustion should behave exactly like hitting max_connections: the acquire waits for a released connection instead of failing the query. Under contention the pool self-clamps to physical port availability; when ports free up, growth resumes toward the full budget. No run-duration cap, no magic fraction, zero dropped samples.

Known pitfalls (from adversarial review — do not land the naive version)

• _wait_for_connection's wake loop calls _can_create_connection(), which only checks the configured ceiling — below the ceiling a woken waiter immediately re-creates and re-fails. Capacity handling must live inside the wait loop (create failure → re-queue; idle-check before parking; no lost-wakeup window between a failed create and parking).
• With zero live pooled connections (e.g. warmup_connections=0 on a port-exhausted host) no release() ever fires _notify_waiter() → waiters hang forever. Needs a zero-live-connections fail-fast (a hang is worse than a dropped sample).
• Errno scoping: divert only capacity errnos (EADDRNOTAVAIL, possibly ENOBUFS/EMFILE); ECONNREFUSED/ENETUNREACH/EHOSTUNREACH/connect timeouts must keep failing fast.
• Blind retry-with-backoff is not a substitute: at a full-range ceiling the losing connects keep losing until a ~60 s TIME_WAIT expires; the waiter queue (woken by releases) is the correct wait condition.
• Unit test shape: mock loop.create_connection to raise errno.EADDRNOTAVAIL while the pool is below its configured ceiling; assert the acquire is served after a release, and assert the zero-live-connections case fails fast instead of hanging.

Minor related item

The budget multiplies by distinct endpoints (config.py), but each worker serves endpoint_urls[worker_id % n] — if endpoints outnumber workers, inactive endpoints inflate the cap.

Interim guidance

Until this lands: for co-located / low-RTT single-endpoint max_throughput runs, set an explicit max_connections below the ephemeral range (e.g. 16384 on the default Linux range) — validated to produce zero failures with no throughput regression.

Trace the connection-pool acquire path, especially _wait_for_connection and _can_create_connection, and review config.py for the endpoint-based budget. Use a mocked loop.create_connection that raises EADDRNOTAVAIL to verify an acquire is served after release and that zero live connections fail fast rather than hanging. Done means capacity errnos wait like max_connections while other connection errors still fail fast.