pingcap / pingcap/tidb

Orphan pessimistic locks from a crashed tidb-server block conflicting transactions for ~20s (ManagedLockTTL), with no way to shorten or clean up proactively

Open
#70,268 1 comment 0 reactions 0 assignees View on GitHub
contribution
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Problem

When a tidb-server instance crashes (kill -9 / OOM / hardware failure), the pessimistic locks held by its in-flight transactions remain on TiKV until their TTL expires. The TTL is `ManagedLockTTL`, a **compile-time constant of 20s** in client-go (`txnkv/transaction/2pc.go`), kept alive by the TTL manager heartbeat while the owner is alive. After the owner dies, every conflicting transaction from surviving TiDB instances is blocked for the full residual TTL (~20s for short transactions).

For contended workloads this means up to ~20 seconds of zero throughput on the affected rows after a single tidb-server crash, and there is currently **no configuration or operational mechanism to shorten it**: the `pessimistic-txn.ttl` config was removed in v4.0 (#13112), and neither TiKV nor PD can proactively clean up locks of a dead TiDB instance because lock records carry no owner-instance information.

### 1. Minimal reproduce step

```sql
create table t (id int primary key, v bigint);
insert into t values (1,0),(2,0),(3,0),(4,0),(5,0);
```

- Two tidb-server instances (e.g. `tiup playground --db 2 --kv 1 --pd 1`), pessimistic mode (default).
- 300 client connections (150 per instance), each looping:
`BEGIN; UPDATE t SET v=v+1 WHERE id=; UPDATE t SET v=v+1 WHERE id=; COMMIT;`
(two distinct random rows out of the 5, always locked in ascending id order, so no deadlocks).
- While the load is running, `kill -9` one tidb-server.

### 2. What did you expect to see?

Transactions on the surviving instance recover within a short, bounded, ideally configurable window — comparable to instance-liveness-detection latency (seconds).

### 3. What did you see instead?

Per-second successful transactions on the surviving instance (all defaults, `innodb_lock_wait_timeout=50`):

```
sec ok/s errors
15 316 0 <- kill -9 the other tidb-server
16~36 0 0 <- zero throughput for ~21s, no errors
37~39 3/179/425 <- queue drain
40 660 <- fully recovered
```

Notes from the reproduction:

- The zero window length is independent of client concurrency (12 vs 300 connections: both ~20–21s) — it is determined purely by the lock TTL.
- With `innodb_lock_wait_timeout=20` (≤ window), all 150 waiters on the surviving node additionally fail with error 1205 in a burst at the end of the window; with the default 50s no statement fails at all.
- Killed-instance in-memory pessimistic locks do not help here: the locks live on TiKV (which stays healthy), only their owner died.

### 4. What is your TiDB version?

Reproduced on a v7.1.9-based build via tiup playground. The mechanism is unchanged on current master: `ManagedLockTTL = 20000 // 20s` in client-go `txnkv/transaction/2pc.go`, initial lock TTL `elapsed + ManagedLockTTL`, heartbeat every TTL/2.

## Root cause

1. Percolator lock records only contain `start_ts` / `primary` / `ttl` — **no owner-instance identity**, so no component can implement "instance X died → resolve its locks". The only liveness signal is the TTL itself.
2. Waiters poll the primary via `CheckTxnStatus`; before TTL expiry they can only keep waiting. GC's resolve-locks path only applies below the GC safepoint and cannot help on a seconds timescale.
3. Interestingly, the original Percolator design did include ownership (worker id in the lock + Chubby session liveness, allowing immediate cleanup with wall-time TTL only as a fallback); this half was dropped in the TiDB implementation.

## Possible improvements (in increasing effort)

1. **Make the pessimistic lock TTL configurable again** (a bounded range, e.g. 3–30s, as a TiDB config/sysvar propagated to client-go). #13112 removed the config item due to rolling-upgrade value-range issues, but the underlying capability still exists. Latency-sensitive deployments could trade heartbeat traffic for a much smaller crash blast radius.
2. **Instance-liveness-based lock resolution without changing the lock format**: PD already hands out TSO in per-client batches, so PD could maintain an "instance → ts-range" mapping and expose "owner of start_ts". Combined with the existing TiDB instance registration in PD/etcd (session TTL of a few seconds), surviving components could resolve locks of a confirmed-dead instance immediately — the same force-expire path GC's `BatchResolveLocks` already uses.
3. **Carry owner-instance identity in the lock record** (kvproto field + client-go populating it + a TiKV-side or TiDB-side janitor correlating with PD instance liveness) — the cleanest fix, equivalent to what the original Percolator paper and other systems (e.g. OceanBase's scheduler-liveness-based participant GC) do. This would bound crash-caused lock-wait windows by liveness-detection latency instead of a fixed 20s TTL.

Happy to provide the load generator and full experiment data (12/300 concurrency, timeout variations) if useful.

Contributor guide

Open the contributing guide

Research direction

Start with client-go's txnkv/transaction/2pc.go, focusing on ManagedLockTTL, the initial lock TTL, and the heartbeat interval. Run the supplied two-instance SQL reproduction to confirm the crash and recovery window. Done requires selecting and specifying one of the proposed TTL or instance-liveness approaches, then validating that surviving transactions recover within the intended bound.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.