During region lifecycle changes (merge / leader transfer / peer destroy), TiKV resolved-ts scans frequently report “resolved_ts scan get snapshot failed: scan task cancelled”.
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
Key Scenario: Continuous writes + high-frequency DDL operations simultaneously trigger region merge/split, leader migration, peer destroy/shutdown mailbox, and other lifecycle changes. Subsequently, TiKV logs show numerous "resolved_ts scan get snapshot failed" errors with the cause "scan task cancelled".
Environment
OS: Ubuntu 22.04
TiDB: 9.0.0 (same version across cluster nodes)
Deployment: Docker, multi-node (tidb1-4); initial anomaly detected in containers/tidb1/tikv.log
Load and Triggering Methods
Two types of loads run concurrently:
- Continuous writes (arbitrary write stress tests, e.g., sysbench oltp_write_only / custom write programs)
- High-frequency DDL loops (e.g., repeated CREATE/DROP/ALTER/TRUNCATE operations)
Observation Points
Monitored in the TiKV logs of node tidb1:
File: containers/tidb1/tikv.log
Critical log entries: resolved_ts scan get snapshot failed, often accompanied by scan task cancelled
### 2. What did you expect to see? (Required)
During frequent region changes (leader migration / merge / peer destruction), if resolved-ts scan tasks are indeed canceled, we hope:
This “expected cancellation” should not flood the log with WARN-level messages (it should be downgraded to INFO/DEBUG or rate-limited); and
Logs should include sufficient context (at minimum region_id / store_id / cancellation reason source, e.g., region shutdown, deregister observe, mailbox closed, not leader, etc.) to determine if attention is required; and
Resolved-TS should continue progressing normally (without getting stuck on certain regions for extended periods).
### 3. What did you see instead (Required)
Numerous WARN messages appear in containers/tidb1/tikv.log (first occurring around line 6600):
[2026/01/19 18:29:53.090 +00:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other(\”[components/resolved_ts/src/scanner.rs:192]: scan task cancelled\“)”] [thread_id=603]
Moreover, within the context surrounding the initial failure point (captured as “approximately 120 lines before and after the first failure point”), distinct region lifecycle change events are observable, including but not limited to:
peer destroy itself / shutdown mailbox
leader transfer + role change + deregister observe region
merge-related: CommitMerge / asking delegate to stop / merge finished / starts destroy
Below is critical context near the first failure point:
[2026/01/19 18:29:53.004 +00:00] [INFO] [peer.rs:1521] [“peer destroy itself”] ... [region_id=47] ...
[2026/01/19 18:29:53.004 +00:00] [INFO] [router.rs:283] [“shutdown mailbox”] [region_id=47] ......
[2026/01/19 18:29:53.253 +00:00] [INFO] [peer.rs:3997] [“transfer leader”] ... [region_id=117] ...
[2026/01/19 18:29:53.253 +00:00] [INFO] [raft.rs:1162] [“became follower at term 7”] ... [region_id=117] ...
[2026/01/19 18:29:53.253 +00:00] [INFO] [endpoint.rs:784] [“deregister observe region”] ... [region_id=117] ...
...
[2026/01/19 18:29:53.405 +00:00] [INFO] [apply.rs:2926] [“asking delegate to stop”] [source_region_id=23] [region_id=27] ...
[2026/01/19 18:29:53.407 +00:00] [INFO] [apply.rs:2948] [“execute CommitMerge”] ... [region_id=27] ...
[2026/01/19 18:29:53.410 +00:00] [INFO] [peer.rs:5357] [“merge finished”] ... [region_id=23] ...
[2026/01/19 18:29:53.410 +00:00] [INFO] [peer.rs:4160] [“starts destroy”] ... [region_id=23] ...
[2026/01/19 18:29:53.413 +00:00] [INFO] [peer.rs:1521] [“peer destroy itself”] ... [region_id=23] ...
[2026/01/19 18:29:53.413 +00:00] [INFO] [router.rs:283] [“shutdown mailbox”] [region_id=23] ...
...
[2026/01/19 18:29:53.090 +00:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other(\”[components/resolved_ts/src/scanner.rs:192]: scan task cancelled\“)”] [thread_id=603]
Current pain points/uncertainties:
This WARN itself lacks a region_id (or it's not visible on a single line), making it difficult to directly determine if it's tied to the same region lifecycle event;
Large log volume makes severity assessment difficult (whether it's expected cancellation noise or abnormal behavior in the resolved-ts scanner, such as duplicate scans/improper cancellation propagation/blocked progress);
Confirmation needed on whether it causes resolved-ts progress blockage (potentially impacting stale reads or TiCDC's resolved-ts progress).
### 4. What is your TiDB version? (Required)
TiDB 9.0.0,Ubuntu 22.04(Docker cluster)
Contributor guide
Assessment
This issue has not been assessed yet.