HTTP client: num_workers must be a multiple of the endpoint count for even load — auto-scale workers for multiple endpoints

Observed behavior (reported from disagg/agg deployments)

With multiple frontend/agg servers behind the client (e.g. a 9×DEP8 aggregation setup), num_workers for the HTTP client must be a multiple of the number of endpoints (9, 18, …) or load is skewed:

• num_workers not a multiple of len(endpoints) → some servers receive more requests than others (endpoints with ⌈w/n⌉ workers get proportionally more traffic than those with ⌊w/n⌋).
• num_workers < len(endpoints) → some servers receive no traffic at all and sit idle.

Mechanism

Each worker process is statically pinned to exactly one endpoint at startup:

# worker.py:150
endpoint_url = endpoint_urls[worker_id % len(endpoint_urls)]

so the per-endpoint request share is proportional to how many workers landed on it — there is no request-level balancing across endpoints.

The auto default makes this easy to hit: num_workers=-1 resolves from NUMA topology (min(max(8, numa_cpu_count), 24)), which is not endpoint-aware — e.g. auto→24 workers with 9 endpoints gives 6 endpoints 3 workers and 3 endpoints 2 workers (a 50% traffic difference between servers), and auto→8 with 9 endpoints leaves one server idle.

Suggested fixes (roughly increasing scope)

1. Validate/warn at config time — in HTTPClientConfig._resolve_defaults, warn when num_workers % distinct_endpoints != 0, and warn loudly (or error) when num_workers < distinct_endpoints (silent idle servers invalidate multi-server perf runs).
2. Make the auto default endpoint-aware — when num_workers=-1 and multiple distinct endpoints are configured, round the NUMA-derived count to a multiple of the endpoint count (e.g. nearest multiple, floored at distinct_endpoints).
3. Longer-term: request-level balancing — decouple workers from single endpoints (per-worker pools to all endpoints with least-loaded/round-robin dispatch). Larger change to the worker hot path; only worth it if the static pinning becomes limiting beyond the divisibility issue.

Related

• The max_connections port budget multiplies by distinct endpoints while workers each serve one endpoint — if endpoints outnumber workers, unreachable endpoints inflate the budget (noted in #418).
• Interim guidance for disagg/agg users: set num_workers explicitly to a multiple of the frontend count (e.g. 9 or 18 for 9×DEP8).

Start with worker.py:150 and HTTPClientConfig._resolve_defaults. Trace how worker counts and distinct endpoints are resolved, then confirm which suggested fix maintainers want, since the issue lists multiple scopes. Done should include agreed behavior for uneven worker counts and endpoint-aware defaults, with coverage for fewer workers and non-multiple counts.