TiCDC changefeed stalls during recovery scan: `scan_window` base_ts freezes, regions never initialize
@asddongmen is already working on this.
Since Aug 18, 2026.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
What did you do?
- TiDB cluster with TiCDC redo log
consistent.level=eventualto S3 (minio), mysql sink. - Changefeed filter
test_ticdc.*; workload writes continuously to 4 tables (t0..t3, ~15k rows/s, 4-byte payloads) via sustained concurrent writers;big_t0/big_t1are static. - Create the changefeed (or pause + resume it after a downtime so a large backlog accumulates) with
start_tsat current time, then observe.
What did you expect to see?
The changefeed should bootstrap, initialize the region subscriptions for all tables, reach Working on all dispatchers, advance the checkpoint, and sync DML to the downstream sink while writing redo logs.
What did you see instead?
The changefeed stays normal with error: null but is functionally stuck:
- Checkpoint frozen at the resume point (never advances).
- Downstream sink receives nothing (
ticdc_sink_*counters flat), no DML written. - Redo writer initially writes a burst then also stops.
- 4 of 12 event-store subscriptions initialize (the static
big_t0/big_t1tables); the 8 subscriptions fort0..t3(each covering ~12–13 regions, 25 regions/table) stayInitialized:falsefor 12+ minutes:span_registry.go:333 "subscription client initializes a region too slow"/:337 "subscription client finds a uninitialized slow region", e.g.RegionID:2265, Initialized:false, Created: 09:11:38.
- Add-dispatcher operators stay in the maintainer running queue:
operator_controller.go:406 "operator is still in running queue" ... timeSinceCreated=8m36s(grows unbounded).
- Sink / redo / event-service goroutines are all healthy in goroutine dumps (no deadlock, no blocked mutex) — the pipeline is simply idle because no dispatcher ever becomes
Working.
Key metrics (all from the stuck changefeed):
ticdc_event_service_scan_task_count 263
ticdc_event_service_interrupt_scan_count 256
ticdc_event_service_pending_scan_task_count 0
ticdc_event_service_scan_window_interval ~103.37 (s) # adaptive recovery interval kept growing 5s -> 103s
ticdc_event_service_scan_window_usage_ema 0.21
ticdc_event_service_scan_window_pressure_score 0
ticdc_event_store_subscription_num 12
# scan_window_base_ts frozen, no advance over 60s:
ticdc_event_service_scan_window_base_ts <frozen>
scan_window_base_ts sampled 60s apart was identical (frozen at a timestamp ~38 min behind current time), proving the recovery scan makes zero forward progress while scans are continuously interrupted.
Versions of the cluster
Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
(paste TiDB cluster version here)
Upstream TiKV version (execute tikv-server --version):
(paste TiKV version here)
TiCDC version (execute cdc version):
PR #5956
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.