pingcap / pingcap/ticdc

Some nodes have no incremental scan traffic for a long time when incrementally scanning multiple large tables

Open
#2,165 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ga-optional type/bug
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

When testing the incremental scan scenario of multiple large tables, multiple nodes will have no incremental scan traffic for a period of time. If you look at the logs, you can see this logs for a long timesubscription client finds a uninitialized slow region.

Image
[2025/09/17 05:42:48.588 +00:00] [INFO] [[endpoint.rs:905](http://endpoint.rs:905/)] ["cdc register region"] [downstream_id=DownstreamId(920)] [observe_id=ObserveId(3143)] [req_id=RequestId(49)] [conn_id=ConnId(16)] [region_id=365312] [thread_id=49]
[2025/09/17 05:45:32.092 +00:00] [INFO] [[delegate.rs:1136](http://delegate.rs:1136/)] ["cdc stop observing"] [failed=false] [region_id=365312] [thread_id=49]
[2025/09/17 05:45:32.162 +00:00] [INFO] [[initializer.rs:137](http://initializer.rs:137/)] ["cdc async incremental scan canceled before start"] [conn_id=ConnId(16)] [observe_id=ObserveId(3143)] [downstream_id=DownstreamId(920)] [region_id=365312] [thread_id=165]
[2025/09/17 05:45:32.162 +00:00] [WARN] [[endpoint.rs:977](http://endpoint.rs:977/)] ["cdc initialize fail: Other error [components/cdc/src/[initializer.rs:142](http://initializer.rs:142/)]: scan canceled"] [request_id=RequestId(49)] [conn_id=ConnId(16)] [region_id=365312] [thread_id=165]
[2025/09/17 05:45:32.162 +00:00] [INFO] [[endpoint.rs:713](http://endpoint.rs:713/)] ["cdc deregister"] [deregister="Deregister { deregister: \"downstream\", conn_id: ConnId(16), request_id: RequestId(49), region_id: 365312, downstream_id: DownstreamId(920), err: Some(Other(\"[components/cdc/src/[initializer.rs:142](http://initializer.rs:142/)]: scan canceled\")) }"] [thread_id=49]
[2025/09/17 05:52:16.673 +00:00] [INFO] [[endpoint.rs:905](http://endpoint.rs:905/)] ["cdc register region"] [downstream_id=DownstreamId(3243)] [observe_id=ObserveId(5469)] [req_id=RequestId(42)] [conn_id=ConnId(92)] [region_id=365312] [thread_id=49]
[2025/09/17 06:02:16.682 +00:00] [WARN] [[delegate.rs:630](http://delegate.rs:630/)] ["cdc region scan locks too slow"] [stage=LockTracker::Pending] [elapsed=600.009s] [region_id=365312] [thread_id=49]
[2025/09/17 06:03:17.042 +00:00] [WARN] [[delegate.rs:630](http://delegate.rs:630/)] ["cdc region scan locks too slow"] [stage=LockTracker::Pending] [elapsed=660.369s] [region_id=365312] [thread_id=49]
[2025/09/17 06:04:17.389 +00:00] [WARN] [[delegate.rs:630](http://delegate.rs:630/)] ["cdc region scan locks too slow"] [stage=LockTracker::Pending] [elapsed=720.716s] [region_id=365312] [thread_id=49]
[2025/09/17 06:05:17.747 +00:00] [WARN] [[delegate.rs:630](http://delegate.rs:630/)] ["cdc region scan locks too slow"] [stage=LockTracker::Pending] [elapsed=781.074s] [region_id=365312] [thread_id=49]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the logged flow through components/cdc/src/endpoint.rs, delegate.rs, and initializer.rs, focusing on the uninitialized slow-region and scan-lock warnings. Reproduce the multiple-large-table incremental scan scenario and compare the registration, cancellation, and lock-tracking events. Done means nodes continue receiving incremental scan traffic without prolonged slow-region warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.