feat: make gateway Postgres connection pool size configurable (hardcoded to 10)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 8.7k
- Forks
- 1.3k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 253
Description
Problem Statement
The gateway's Postgres pool is hardcoded to max_connections(10) (crates/openshell-server/src/persistence/postgres.rs), with no config or env path. It's a single pool shared by every DB-backed RPC, so 10 is a hard, client-side ceiling on gateway↔DB concurrency — independent of how large the database is.
In cluster mode with external Postgres (workload.kind=deployment + server.externalDbSecret), a fleet of sandbox supervisors polling GetSandboxConfig / GetInferenceBundle / provider-env easily exceeds 10 concurrent connections. Symptoms:
sqlx::pool::acquireslow-threshold warnings; multi-second acquire latency on top of every DB-backed RPC./readyzbecomes latency-sensitive to pool pressure, since it also acquires from this pool (PostgresStore::ping()). (This surfaced as readiness flapping in our case, but the root cause there was the default readiness-probe timeout being too short to tolerate a slow acquire — operator-tunable — not the pool itself. Noted only because it shows readiness is coupled to pool saturation.)
Scaling the database up doesn't help: the pool never opens more than 10 connections, leaving the extra capacity idle.
Proposed Design
Make the pool options configurable via environment variables, each defaulting to today's value (no change for existing deployments). Every one is read at startup and drops straight into the Helm chart as a container env: value — no image rebuild or config-file edit — consistent with OPENSHELL_DB_URL:
OPENSHELL_DB_MAX_CONNECTIONS— the pool cap (default 10). The primary fix.OPENSHELL_DB_ACQUIRE_SLOW_THRESHOLD_SECS— the sqlx slow-acquire warn threshold (currently the sqlx default of 2s), so operators can tune or quiet that log line independently of the pool size.- Optionally
OPENSHELL_DB_MIN_CONNECTIONSandOPENSHELL_DB_ACQUIRE_TIMEOUT_SECS.
Thread these into the pool builder (Store::connect → PostgresStore::connect → PgPoolOptions). They may also be mirrored in gateway.toml [openshell.gateway.database] (as log_level is today), with the env vars taking precedence.
Operator note: total connections = pool_size × replica_count, which must stay under the server's max_connections.
Agent Investigation
- Single shared pool:
Store::connect(crates/openshell-server/src/lib.rs) builds onePostgresStore/PgPoolat startup. - Hardcoded cap:
PgPoolOptions::new().max_connections(10).connect(url)(persistence/postgres.rs) —connecttakes only the URL, no pool knob. - Shared by all handlers: e.g.
handle_get_sandbox_config(grpc/policy.rs) does several store reads per request, each from the same 10 slots.
Checklist
- I've reviewed existing issues and the architecture docs
- This is a design proposal, not a "please build this" request
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Store::connect in crates/openshell-server/src/lib.rs and follow it into PostgresStore::connect and PgPoolOptions in crates/openshell-server/src/persistence/postgres.rs. Review how OPENSHELL_DB_URL reaches the Helm chart and how gateway.toml database settings are handled. Done means the pool settings are configurable at startup with today's defaults preserved, environment precedence defined, and the total connection guidance documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, rust
- Domain
- backend, database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100