Scale sandbox proxy read throughput: multi-worker uvicorn + shared httpx pools
- Dominant language
- Python
- Stars
- 485
- Forks
- 81
- Avg merge
- 16h 12m
- Merged PRs (30d)
- 8
Description
## What
The sandbox proxy / admin read path is bottlenecked on a single-process event loop, a per-request `httpx.AsyncClient`, a small Ray thread-pool executor, and a fixed DB pool. This raises read throughput by:
- Running the proxy role across multiple uvicorn workers (opt-in via `--workers` / `ROCK_PROXY_WORKERS`), while keeping the admin role and `local`/`test`/`dev` single-process (they own scheduler/Ray singletons and per-process in-memory/fakeredis state).
- Reusing shared httpx connection pools (separate control-plane and data-plane clients) instead of constructing a new `AsyncClient` on every request.
## Why
Under concurrent read load, per-request client construction and the small executor/pool sizes cap effective throughput and waste connections. Multi-worker + pooled clients spread CPU across cores and reuse keep-alive connections, removing the single-loop ceiling without regressing per-pod connection budgets.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the sandbox proxy startup entry point and trace the proxy role's uvicorn configuration, including --workers and ROCK_PROXY_WORKERS. Then trace the proxy request paths and current httpx client construction. Done means opt-in proxy workers and shared control-plane/data-plane clients work without changing admin or local/test/dev single-process behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100