Orphan pessimistic locks from a crashed tidb-server block conflicting transactions for ~20s (ManagedLockTTL), with no way to shorten or clean up proactively
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Problem
When a tidb-server instance crashes (kill -9 / OOM / hardware failure), the pessimistic locks held by its in-flight transactions remain on TiKV until their TTL expires. The TTL is `ManagedLockTTL`, a **compile-time constant of 20s** in client-go (`txnkv/transaction/2pc.go`), kept alive by the TTL manager heartbeat while the owner is alive. After the owner dies, every conflicting transaction from surviving TiDB instances is blocked for the full residual TTL (~20s for short transactions).
For contended workloads this means up to ~20 seconds of zero throughput on the affected rows after a single tidb-server crash, and there is currently **no configuration or operational mechanism to shorten it**: the `pessimistic-txn.ttl` config was removed in v4.0 (#13112), and neither TiKV nor PD can proactively clean up locks of a dead TiDB instance because lock records carry no owner-instance information.
### 1. Minimal reproduce step
```sql
create table t (id int primary key, v bigint);
insert into t values (1,0),(2,0),(3,0),(4,0),(5,0);
```
- Two tidb-server instances (e.g. `tiup playground --db 2 --kv 1 --pd 1`), pessimistic mode (default).
- 300 client connections (150 per instance), each looping:
`BEGIN; UPDATE t SET v=v+1 WHERE id=; UPDATE t SET v=v+1 WHERE id=; COMMIT;`
(two distinct random rows out of the 5, always locked in ascending id order, so no deadlocks).
- While the load is running, `kill -9` one tidb-server.
### 2. What did you expect to see?
Transactions on the surviving instance recover within a short, bounded, ideally configurable window — comparable to instance-liveness-detection latency (seconds).
### 3. What did you see instead?
Per-second successful transactions on the surviving instance (all defaults, `innodb_lock_wait_timeout=50`):
```
sec ok/s errors
15 316 0 <- kill -9 the other tidb-server
16~36 0 0 <- zero throughput for ~21s, no errors
37~39 3/179/425 <- queue drain
40 660 <- fully recovered
```
Notes from the reproduction:
- The zero window length is independent of client concurrency (12 vs 300 connections: both ~20–21s) — it is determined purely by the lock TTL.
- With `innodb_lock_wait_timeout=20` (≤ window), all 150 waiters on the surviving node additionally fail with error 1205 in a burst at the end of the window; with the default 50s no statement fails at all.
- Killed-instance in-memory pessimistic locks do not help here: the locks live on TiKV (which stays healthy), only their owner died.
### 4. What is your TiDB version?
Reproduced on a v7.1.9-based build via tiup playground. The mechanism is unchanged on current master: `ManagedLockTTL = 20000 // 20s` in client-go `txnkv/transaction/2pc.go`, initial lock TTL `elapsed + ManagedLockTTL`, heartbeat every TTL/2.
## Root cause
1. Percolator lock records only contain `start_ts` / `primary` / `ttl` — **no owner-instance identity**, so no component can implement "instance X died → resolve its locks". The only liveness signal is the TTL itself.
2. Waiters poll the primary via `CheckTxnStatus`; before TTL expiry they can only keep waiting. GC's resolve-locks path only applies below the GC safepoint and cannot help on a seconds timescale.
3. Interestingly, the original Percolator design did include ownership (worker id in the lock + Chubby session liveness, allowing immediate cleanup with wall-time TTL only as a fallback); this half was dropped in the TiDB implementation.
## Possible improvements (in increasing effort)
1. **Make the pessimistic lock TTL configurable again** (a bounded range, e.g. 3–30s, as a TiDB config/sysvar propagated to client-go). #13112 removed the config item due to rolling-upgrade value-range issues, but the underlying capability still exists. Latency-sensitive deployments could trade heartbeat traffic for a much smaller crash blast radius.
2. **Instance-liveness-based lock resolution without changing the lock format**: PD already hands out TSO in per-client batches, so PD could maintain an "instance → ts-range" mapping and expose "owner of start_ts". Combined with the existing TiDB instance registration in PD/etcd (session TTL of a few seconds), surviving components could resolve locks of a confirmed-dead instance immediately — the same force-expire path GC's `BatchResolveLocks` already uses.
3. **Carry owner-instance identity in the lock record** (kvproto field + client-go populating it + a TiKV-side or TiDB-side janitor correlating with PD instance liveness) — the cleanest fix, equivalent to what the original Percolator paper and other systems (e.g. OceanBase's scheduler-liveness-based participant GC) do. This would bound crash-caused lock-wait windows by liveness-detection latency instead of a fixed 20s TTL.
Happy to provide the load generator and full experiment data (12/300 concurrency, timeout variations) if useful.
Contributor guide
Research direction
Start with client-go's txnkv/transaction/2pc.go, focusing on ManagedLockTTL, the initial lock TTL, and the heartbeat interval. Run the supplied two-instance SQL reproduction to confirm the crash and recovery window. Done requires selecting and specifying one of the proposed TTL or instance-liveness approaches, then validating that surviving transactions recover within the intended bound.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100