lablup / lablup/backend.ai

Expose backend connection pool limit for App Proxy Worker

Open
#13,408 1 comment 0 reactions 0 assignees Claimed by @rapsealk View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

### Main idea

## Summary

`HTTPBackend` in App Proxy Worker creates its backend HTTP client without
specifying aiohttp's `limit` parameter, so the default of 100 concurrent
connections per session applies. This becomes a bottleneck under high
concurrency and is not adjustable via configuration.

Confirmed present on `main` at the time of writing.

## Current Behavior

In `src/ai/backend/appproxy/worker/proxy/backend/http.py`:

```python
self.client_pool = ClientPool(
partial(
tcp_client_session_factory,
timeout=client_timeout,
auto_decompress=False,
),
cleanup_interval_seconds=cleanup_interval,
)
```

`tcp_client_session_factory` (in
`src/ai/backend/common/clients/http_client/client_pool.py`) defaults to
`limit: int = 100`, which is passed to `aiohttp.TCPConnector`.

Since `limit` is not specified at the call site, each `ClientSession` created
per `ClientKey` gets a `TCPConnector` capped at 100 connections. With N
replicas (distinct endpoints), the effective ceiling is 100×N, which is still
easily exceeded under bursty load.

`client_pool_cleanup_interval` is exposed in `worker/config.py`, but there is
no corresponding option for the pool size.

## Observation

Benchmarked with `vllm bench serve` (ShareGPT dataset, 500 prompts,
`--request-rate inf`, `--seed 42`) against a vLLM model service.

**Via App Proxy (2 replicas):**
- 55 / 500 requests failed
- `aiohttp.client_exceptions.ConnectionTimeoutError: Connection timeout to
host http://:/v1/completions`, surfaced to the client as
Gateway Timeout
- Stack trace points to
`aiohttp/connector.py:_wait_for_available_connection`, i.e. requests
queueing for a free connection inside the proxy's connector rather than
being rejected by the backend

**Directly to a single vLLM replica (bypassing App Proxy), identical
workload:**
- 0 / 500 requests failed
- Per-request latency was notably worse (TPOT p99 799ms vs 58ms via proxy),
as expected from a single replica absorbing the entire load

With 2 replicas the effective connection ceiling would be ~200, well below
the 500 concurrent requests offered. The single replica sustained the full
500 without failures despite heavier per-replica load, which suggests the
failures originate from the proxy layer's connection pool rather than
backend saturation.

## Proposal

Expose the pool size as a configurable field (e.g.
`backend_connection_pool_limit`) in `worker/config.py`, following the same
pattern as `client_pool_cleanup_interval`, and pass it through to
`tcp_client_session_factory`.

I understand running multiple workers is the intended scaling path per the
App Proxy architecture, but a single tunable for the connection pool would
let operators absorb burst traffic without changing the deployment topology.
The change looks small in scope (one config field plus the call site).

### Alternative ideas

- Set `limit=0` (unlimited) at the call site by default, on the grounds that
a reverse proxy should not impose its own connection cap below what the
backends can handle. This requires no new config field, but removes the
safety ceiling entirely, so an explicit opt-in config might be safer.
- Expose `limit_per_host` alongside `limit`. Since sessions are keyed per
endpoint (`ClientKey`), `limit` is the effective knob today, but exposing
both would match aiohttp's connector semantics and be more future-proof.

### Anything else?

## Environment

- Backend.AI 26.4.4 (`backend.ai-appproxy-worker 26.4.4`)
- Installed via Docker Containers method
- Model service: vLLM 0.23.0, 2 replicas on a single agent node

I'd be glad to submit a PR if this direction sounds reasonable — please let
me know if you'd prefer a different approach.

Contributor guide

Open the contributing guide

Research direction

Start with src/ai/backend/appproxy/worker/config.py and the HTTPBackend call site in src/ai/backend/appproxy/worker/proxy/backend/http.py, then read tcp_client_session_factory in src/ai/backend/common/clients/http_client/client_pool.py. Done means the pool limit is configurable through the worker settings and reaches the backend HTTP client factory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.