pingcap / pingcap/tidb

Stale Lock Remains After Async Commit Transaction Completion, Not Resolved by GC

Open
#65,757 7 comments 0 reactions 0 assignees View on GitHub
affects-7.1 affects-7.5 affects-8.1 affects-8.5 contribution severity/critical sig/transaction type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

### What did you do?

Encountered a persistent lock error during key scanning operations in production.

### What did you expect to see?

Normal key scan operations without lock errors. If a transaction is committed, all associated locks should be cleaned up either during the commit process or by GC.

### What did you see instead?

**Error encountered:**

```
[err_code=KV:Unknown] [err="Transaction error key is locked (backoff or cleanup)
primary_lock: 74800000000000012F5F6980000000000000010170726F6A6563743AFF3131323138313200FE010FC01002021B0F5BFF0E8B10020FC00EFBFF0E600FEA00000000FB01028C027E0EB90FC0FF0F820F5B027E023DFF027E0E2B0E290E2BFF0E2E02210E2A0E2BFF02210E2B0E29027EFF022F027E10020F82FF027E023D027E0E2BFF0E290E2B0E2F0221FF0E290E2A02210E2AFF0E32027E028D0000FD
lock_version: 463694439080781172
key: 74800000000000012F5F728000000069BFA664
lock_ttl: 3001
txn_size: 1
use_async_commit: true
min_commit_ts: 463694439473741825"]
```

**Analysis:**

The lock is from a very old transaction (transaction timestamp: `463694439080781172`):

```sql
SELECT tidb_parse_tso(463694439080781172);
+------------------------------------+
| tidb_parse_tso(463694439080781172) |
+------------------------------------+
| 2026-01-19 20:18:28.846000 |
+------------------------------------+
```

**Evidence of Issue:**

1. **Secondary key still has lock** (verified with `tikv-ctl`):

```bash
$ tikv-ctl scan --from 'zt\200\000\000\000\000\000\001\377/_r\200\000\000\000i\377\277\246d\000\000\000\000\000\372' --limit 1 --show-cf default,write,lock
key: zt\200\000\000\000\000\000\001\377/_r\200\000\000\000i\377\277\246d\000\000\000\000\000\372
lock cf value: start_ts: 463694439080781172 primary: 74800000000000012F5F6980000000000000010170726F6A6563743AFF3131323138313200FE010FC01002021B0F5BFF0E8B10020FC00EFBFF0E600FEA00000000FB01028C027E0EB90FC0FF0F820F5B027E023DFF027E0E2B0E290E2BFF0E2E02210E2A0E2BFF02210E2B0E29027EFF022F027E10020F82FF027E023D027E0E2BFF0E290E2B0E2F0221FF0E290E2A02210E2AFF0E32027E028D0000FD short_value: 800004000000020304050A0019004000480072745F6D65747269637370726F6A6563743A313132313831327B2266726F6D223A22323032352D31322D3230222C22746F223A22323032362D30312D3139227D0000009944E7B819
```

2. **Primary key shows transaction `463694439080781172` is COMMITTED**:

```bash
$ tikv-ctl scan --from 'zt\200\000\000\000\000\000\001\377/_i\200\000\000\000\000\377\000\000\001\001proj\377ect:\377112\3771812\000\376\001\017\377\300\020\002\002\033\017[\377\377\016\213\020\002\017\300\016\373\377\377\016`\017\352\000\000\000\377\000\373\001\002\214\002~\016\377\271\017\300\377\017\202\017[\377\002~\002=\377\002~\016\377+\016)\016+\377\016.\377\002!\016*\016+\377\002\377!\016+\016)\002~\377\377\002/\002~\020\002\017\202\377\377\002~\002=\002~\016\377+\377\016)\016+\016/\377\002!\377\016)\016*\002\377!\016*\377\0162\002~\377\002\215\000\000\375\000\000\000\374' --limit 1 --show-cf default,write,lock
key: zt\200\000\000\000\000\000\001\377/_i\200\000\000\000\000\377\000\000\001\001proj\377ect:\377112\3771812\000\376\001\017\377\300\020\002\002\033\017[\377\377\016\213\020\002\017\300\016\373\377\377\016`\017\352\000\000\000\377\000\373\001\002\214\002~\016\377\271\017\300\377\017\202\017[\377\002~\002=\377\002~\016\377+\016)\016+\377\016.\377\002!\016*\016+\377\002\377!\016+\016)\002~\377\377\002/\002~\020\002\017\202\377\377\002~\002=\002~\016\377+\377\016)\016+\016/\377\002!\377\016)\016*\002\377!\016*\377\0162\002~\377\002\215\000\000\375\000\000\000\374
write cf value: start_ts: 463694439080781172 commit_ts: 463694439080783335 short_value: 088000030000000203040A001900400072745F6D65747269637370726F6A6563743A313132313831327B2266726F6D223A22323032352D31322D3230222C22746F223A22323032362D30312D3139227D0000000069BFA664
```

### Root Cause Questions

This reveals an inconsistent transaction state:

1. **Partial Lock Resolution Issue**: How can the primary key of an async commit transaction be successfully committed (with `commit_ts: 463694439080783335` in write CF) while a secondary key from the same transaction still has its lock in the lock CF? This violates transaction atomicity guarantees.

2. **GC Failure to Clean Stale Locks**: Why hasn't GC resolved this stale lock? The transaction was committed days ago (2026-01-19), yet the lock persists. What conditions prevent GC from cleaning up locks from committed async commit transactions?

3. How can we prevent this from happening?

### Impact

- **Severity**: High - causes query failures and blocks key scans
- **Data Consistency**: Transaction atomicity is violated (partial commit state visible)
- **Operational**: Stale locks accumulate over time and are not cleaned by GC

### Versions

**TiDB Version:**

8.5.3

**Deployment:**
Bare-metal, 400 TiKV nodes, with encryption at rest enabled.

### Additional Context

- Transaction uses `use_async_commit: true`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.