feat: add proxy HTTP connection pool monitoring
- Dominant language
- Python
- Stars
- 485
- Forks
- 81
- Avg merge
- 16h 12m
- Merged PRs (30d)
- 8
Description
## Background
The `SandboxProxyService` uses an httpx connection pool (`proxy`) for data-plane streaming/SSE requests to downstream Rocklets. Currently there is no observability into this pool's utilization — when the pool saturates (hits `max_connections`), requests silently queue with no visibility in monitoring.
## Problem
In production we observed:
- Some workers' proxy pools hitting the 100-connection cap with hundreds of requests queued
- Uneven distribution across workers due to long-lived SSE connections
- No way to detect pool saturation from monitoring dashboards
## Proposal
1. **Add HTTP pool metrics** — periodically (every 10s) report `active_connections`, `idle_connections`, and `pending_requests` (truly queued, waiting for a free connection) as OpenTelemetry gauges
2. **Increase proxy pool limits** — raise `max_connections` from 100 to 2000 and `max_keepalive_connections` from 50 to 100 to reduce queueing under load
3. **Fix empty error messages** — `handle_exceptions` decorator logs empty string when `str(e)` is empty, fall back to `repr(e)`
## Files Changed
- `rock/admin/metrics/constants.py` — new gauge constants
- `rock/admin/metrics/monitor.py` — register gauges
- `rock/utils/http_pool.py` — `get_pool_stats()` method
- `rock/sandbox/service/sandbox_proxy_service.py` — APScheduler periodic collection
- `rock/config.py` — increase proxy pool limits
- `rock/common/exception.py` — fix empty error messages
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with rock/utils/http_pool.py and rock/sandbox/service/sandbox_proxy_service.py to trace the proxy pool and its periodic collection path, then review gauge registration in rock/admin/metrics/constants.py and rock/admin/metrics/monitor.py. Check rock/config.py for the proxy limits and rock/common/exception.py for empty exception messages. Done means the three pool gauges are reported every 10 seconds, limits are updated, and empty messages fall back to repr(e).
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100