cockroachdb / cockroachdb/cockroach
sql/catalog/lease: orphaned dead-session lease cleanup doesn't scale to large backlogs
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Problem
`system.lease` accumulates rows from dead sqlliveness sessions — when a node exits without releasing its leases (crash, OOM, SIGKILL, abrupt pod restart), it leaves a lease row for every descriptor it held. The only thing that removes these is the startup cleanup `deleteOrphanedLeasesFromStaleSession` (`DeleteOrphanedLeases`, run once per node at SQL-server start). With a large backlog it can't keep up, and on a mass restart it becomes a thundering herd on `system.lease`.
Three compounding factors:
1. **Every node runs it, at startup** — a mass/rolling restart fires N cleanups at once.
2. **All nodes target the same rows.** `SELECT DISTINCT session_id … LIMIT 50` returns the same dead sessions in the same order on every node, so they all issue the same `DELETE` and contend on the same keys — effectively serial regardless of node count, plus retry overhead.
3. **Each delete is a full scan with a tiny batch.** `system.lease` PK is `(crdb_region, desc_id, version, session_id)` with no secondary index on `session_id`, so `DELETE … WHERE session_id=X LIMIT 50` scans to find 50 rows; `LIMIT 50` × `MaxRetries 10` clears only a few hundred rows/session/startup, then stops.
Net: heavy, sustained write/write contention on a small, cluster-critical table while the backlog barely shrinks — so the bloat persists and the herd re-forms on the next restart. (Observed on v25.2.17; the cleanup logic is unchanged on current master.)
## Observational data (one field occurrence)
- `system.lease`: ~8.8M rows, ~99.9% from dead sessions (~1,954 distinct dead sessions vs. ~51 live).
- ~51 concurrent `internal-delete-orphaned-leases-by-session` deletes running, **34 deleting the same `session_id`** at once.
- Locks on the `system.lease` range held/queued for **9+ hours**.
- Back-of-envelope: at `LIMIT 50` + a full scan per delete (~seconds each under contention), the backlog is ~days to drain and never finishes in a single restart. For contrast, a single predicate-style delete at `LIMIT 50000` drains the same backlog in ~minutes.
## Possible directions (seeds — owner decides)
Intentionally left open. Two independent levers to consider:
1. **Per-op efficiency — the delete shape.** A predicate-in-PK-order delete (e.g. `DELETE … WHERE NOT EXISTS (SELECT 1 FROM system.sqlliveness …) LIMIT `, looped) is fast *without* an index, since dead rows are dense during a backlog and a large batch amortizes the scan over thousands of rows instead of 50. (A secondary index on `session_id` would also speed the current per-session delete, but adds write amplification on a hot, write-heavy table.)
2. **Who runs it — the herd/trigger.** E.g. a single coordinator / periodic singleton job, or partition the work by a PK prefix (`desc_id` range). Partitioning by `session_id` hash is a trap — the hash isn't indexed, so each node would full-scan for its slice (N× read load).
Worth noting: today there is **no continuous reaper** of orphaned dead-session leases — they're only swept at startup, best-effort — so any ungraceful node churn silently accumulates until a restart. A periodic approach would close that root gap, not just the herd.
## Related
Separate issue: a `DROP`/schema-change `WaitForNoVersion` hang when the dead-session set exceeds the sqlliveness `is_alive` cache (cached liveness mis-counts dead sessions as alive) — cockroachdb/cockroach#172036. Same backlog is the precondition, but a different failure mode and fix.
Jira issue: CRDB-65233
Epic CRDB-65516
Contributor guide
Research direction
Start at the DeleteOrphanedLeases entry point and its deleteOrphanedLeasesFromStaleSession startup cleanup, then inspect how system.lease and system.sqlliveness are queried. Compare the cleanup trigger and delete shape against the observed backlog and contention. Done means orphaned leases are removed efficiently without every node repeatedly targeting the same sessions, with behavior validated under a large backlog.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100