Some nodes have no incremental scan traffic for a long time when incrementally scanning multiple large tables
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
When testing the incremental scan scenario of multiple large tables, multiple nodes will have no incremental scan traffic for a period of time. If you look at the logs, you can see this logs for a long timesubscription client finds a uninitialized slow region.
[2025/09/17 05:42:48.588 +00:00] [INFO] [[endpoint.rs:905](http://endpoint.rs:905/)] ["cdc register region"] [downstream_id=DownstreamId(920)] [observe_id=ObserveId(3143)] [req_id=RequestId(49)] [conn_id=ConnId(16)] [region_id=365312] [thread_id=49]
[2025/09/17 05:45:32.092 +00:00] [INFO] [[delegate.rs:1136](http://delegate.rs:1136/)] ["cdc stop observing"] [failed=false] [region_id=365312] [thread_id=49]
[2025/09/17 05:45:32.162 +00:00] [INFO] [[initializer.rs:137](http://initializer.rs:137/)] ["cdc async incremental scan canceled before start"] [conn_id=ConnId(16)] [observe_id=ObserveId(3143)] [downstream_id=DownstreamId(920)] [region_id=365312] [thread_id=165]
[2025/09/17 05:45:32.162 +00:00] [WARN] [[endpoint.rs:977](http://endpoint.rs:977/)] ["cdc initialize fail: Other error [components/cdc/src/[initializer.rs:142](http://initializer.rs:142/)]: scan canceled"] [request_id=RequestId(49)] [conn_id=ConnId(16)] [region_id=365312] [thread_id=165]
[2025/09/17 05:45:32.162 +00:00] [INFO] [[endpoint.rs:713](http://endpoint.rs:713/)] ["cdc deregister"] [deregister="Deregister { deregister: \"downstream\", conn_id: ConnId(16), request_id: RequestId(49), region_id: 365312, downstream_id: DownstreamId(920), err: Some(Other(\"[components/cdc/src/[initializer.rs:142](http://initializer.rs:142/)]: scan canceled\")) }"] [thread_id=49]
[2025/09/17 05:52:16.673 +00:00] [INFO] [[endpoint.rs:905](http://endpoint.rs:905/)] ["cdc register region"] [downstream_id=DownstreamId(3243)] [observe_id=ObserveId(5469)] [req_id=RequestId(42)] [conn_id=ConnId(92)] [region_id=365312] [thread_id=49]
[2025/09/17 06:02:16.682 +00:00] [WARN] [[delegate.rs:630](http://delegate.rs:630/)] ["cdc region scan locks too slow"] [stage=LockTracker::Pending] [elapsed=600.009s] [region_id=365312] [thread_id=49]
[2025/09/17 06:03:17.042 +00:00] [WARN] [[delegate.rs:630](http://delegate.rs:630/)] ["cdc region scan locks too slow"] [stage=LockTracker::Pending] [elapsed=660.369s] [region_id=365312] [thread_id=49]
[2025/09/17 06:04:17.389 +00:00] [WARN] [[delegate.rs:630](http://delegate.rs:630/)] ["cdc region scan locks too slow"] [stage=LockTracker::Pending] [elapsed=720.716s] [region_id=365312] [thread_id=49]
[2025/09/17 06:05:17.747 +00:00] [WARN] [[delegate.rs:630](http://delegate.rs:630/)] ["cdc region scan locks too slow"] [stage=LockTracker::Pending] [elapsed=781.074s] [region_id=365312] [thread_id=49]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the logged flow through components/cdc/src/endpoint.rs, delegate.rs, and initializer.rs, focusing on the uninitialized slow-region and scan-lock warnings. Reproduce the multiple-large-table incremental scan scenario and compare the registration, cancellation, and lock-tracking events. Done means nodes continue receiving incremental scan traffic without prolonged slow-region warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100