dotCMS / dotCMS/core

ops(rds): Apply server-side session timeouts as backstop for connection leaks

Open
#34,922 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dotCMS : Technical Debt stale Team : Enablement Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Split from #34832

This issue extracts the server-side PostgreSQL timeout configuration from the omnibus #34832 into standalone operational work. These are RDS parameter changes applied per-role — no code deployment required.


Description

Server-side timeouts provide a backstop that catches any connection leak regardless of the application-level fix. These prevent leaked connections from persisting indefinitely and consuming RDS connection slots.

Changes

Apply per-role (immediate, no restart required). Takes effect for new sessions only:

-- Backstop for any query running longer than 10 minutes.
-- Covers Mode A (slow/locked query) at the database level.
ALTER ROLE dotcms_user SET statement_timeout = '600000';

-- Backstop for connections stuck idle inside an open transaction for
-- longer than 30 minutes. Conservative given the known indexing-in-transaction
-- anti-pattern (#34833) — genuine publish transactions do not sit idle for 30 min.
ALTER ROLE dotcms_user SET idle_in_transaction_session_timeout = '1800000';

-- Backstop for connections that have fully escaped pool management.
-- 2 hours ensures pooled connections cycling at maxLifetime=30min are
-- never affected.
ALTER ROLE dotcms_user SET idle_session_timeout = '7200000';

Verify settings applied:

SELECT rolname, rolconfig FROM pg_roles WHERE rolname = 'dotcms_user';

Applies to all RDS instances

These settings should be applied to all shared RDS instances across all regions:

  • k8s-comm-1-green (Frankfurt)
  • US East prod RDS
  • Canada prod RDS
  • Sydney prod RDS

Note: Apply per role, not globally, to avoid affecting superuser and monitoring connections.

Required timeout ordering

idle_session_timeout  (server, 2h)
    > maxLifetime     (HikariCP, 30 min after #34921)
    > idleTimeout     (HikariCP, 5 min)
    > keepaliveTime   (HikariCP, 2 min — after tcpKeepAlive work in #34832)
    > NAT idle timeout (AWS, ~350s)

Future tightening (blocked)

Setting Current proposed Target Blocked by
idle_in_transaction_session_timeout 30 min 5–10 min #34833 (ES calls inside @WrapInTransaction)
idle_session_timeout 2 hours 1 hour Confirm keepaliveTime working in prod (#34832)

Acceptance Criteria

  • statement_timeout, idle_in_transaction_session_timeout, idle_session_timeout applied per-role on all production RDS instances
  • Settings verified via SELECT rolname, rolconfig FROM pg_roles
  • No false positives (legitimate operations not killed)
  • Document applied values in runbook

Links

  • Extracted from: #34832
  • Epic: #34837
  • Blocked by (for tightening): #34833

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the listed production RDS instances and the per-role PostgreSQL ALTER ROLE statements; no repository file is named. Apply the three settings to dotcms_user, then verify them with the provided pg_roles query on every instance. Done means all acceptance criteria are met, including recording the values in the runbook and confirming no legitimate operations are killed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, postgresql
Domain
cloud, databases, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.