ops(rds): Apply server-side session timeouts as backstop for connection leaks
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Split from #34832
This issue extracts the server-side PostgreSQL timeout configuration from the omnibus #34832 into standalone operational work. These are RDS parameter changes applied per-role — no code deployment required.
Description
Server-side timeouts provide a backstop that catches any connection leak regardless of the application-level fix. These prevent leaked connections from persisting indefinitely and consuming RDS connection slots.
Changes
Apply per-role (immediate, no restart required). Takes effect for new sessions only:
-- Backstop for any query running longer than 10 minutes.
-- Covers Mode A (slow/locked query) at the database level.
ALTER ROLE dotcms_user SET statement_timeout = '600000';
-- Backstop for connections stuck idle inside an open transaction for
-- longer than 30 minutes. Conservative given the known indexing-in-transaction
-- anti-pattern (#34833) — genuine publish transactions do not sit idle for 30 min.
ALTER ROLE dotcms_user SET idle_in_transaction_session_timeout = '1800000';
-- Backstop for connections that have fully escaped pool management.
-- 2 hours ensures pooled connections cycling at maxLifetime=30min are
-- never affected.
ALTER ROLE dotcms_user SET idle_session_timeout = '7200000';
Verify settings applied:
SELECT rolname, rolconfig FROM pg_roles WHERE rolname = 'dotcms_user';
Applies to all RDS instances
These settings should be applied to all shared RDS instances across all regions:
k8s-comm-1-green(Frankfurt)- US East prod RDS
- Canada prod RDS
- Sydney prod RDS
Note: Apply per role, not globally, to avoid affecting superuser and monitoring connections.
Required timeout ordering
idle_session_timeout (server, 2h)
> maxLifetime (HikariCP, 30 min after #34921)
> idleTimeout (HikariCP, 5 min)
> keepaliveTime (HikariCP, 2 min — after tcpKeepAlive work in #34832)
> NAT idle timeout (AWS, ~350s)
Future tightening (blocked)
| Setting | Current proposed | Target | Blocked by |
|---|---|---|---|
idle_in_transaction_session_timeout |
30 min | 5–10 min | #34833 (ES calls inside @WrapInTransaction) |
idle_session_timeout |
2 hours | 1 hour | Confirm keepaliveTime working in prod (#34832) |
Acceptance Criteria
-
statement_timeout,idle_in_transaction_session_timeout,idle_session_timeoutapplied per-role on all production RDS instances - Settings verified via
SELECT rolname, rolconfig FROM pg_roles - No false positives (legitimate operations not killed)
- Document applied values in runbook
Links
- Extracted from: #34832
- Epic: #34837
- Blocked by (for tightening): #34833
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the listed production RDS instances and the per-role PostgreSQL ALTER ROLE statements; no repository file is named. Apply the three settings to dotcms_user, then verify them with the provided pg_roles query on every instance. Done means all acceptance criteria are met, including recording the values in the runbook and confirming no legitimate operations are killed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, postgresql
- Domain
- cloud, databases, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100