pingcap / pingcap/ticdc

CDC OOM When There Are 60k Dispatchers with Syncpoint Enabled and Many Truncate Table DDLs

Open
#4,842 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

severity/moderate type/bug
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

During incremental catch-up, due to the need to process syncpoints/DDLs accumulated over a period of time, a large number of dispatcherStatusRequest are repeatedly sent for processing.
Then, because there are a relatively large number of dispatchers, the maintainer's handling speed cannot keep up with their resending speed. Eventually, this overwhelms the system, resulting in an OOM situation.

Image
➜  log grep "resend task periodic resend" cdc-2026-04-15T18-54-13.836.log | wc -l
322481
➜  log tail cdc-2026-04-15T18-54-13.836.log
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=99651912610150768516142891601317547648] [message="ID:<high:6142891601317547648 low:9965191261015076851 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=56387882334628808071329043824189030809] [message="ID:<high:1329043824189030809 low:5638788233462880807 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]
[2026/04/15 18:54:13.836 +08:00] [INFO] [event_collector.go:447] ["release dispatcher memory in DS"] [dispatcherID=137131577023193707788927581790319772544]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=1659228167089718475513856676581264965551] [message="ID:<high:13856676581264965551 low:16592281670897184755 > state:<IsBlocked:true BlockTs:465633269514240000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=10]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=177467660874281160245570627732352186044] [message="ID:<high:5570627732352186044 low:17746766087428116024 > state:<IsBlocked:true BlockTs:465633269514240000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=10]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=35515166653188255502549568495179157148] [message="ID:<high:2549568495179157148 low:3551516665318825550 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]
[2026/04/15 18:54:13.836 +08:00] [INFO] [event_collector.go:447] ["release dispatcher memory in DS"] [dispatcherID=54953054923390040866848125201506941597]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=131778944883591698812614583334147500163] [message="ID:<high:12614583334147500163 low:1317789448835916988 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]
[2026/04/15 18:54:13.836 +08:00] [INFO] [event_collector.go:447] ["release dispatcher memory in DS"] [dispatcherID=470363268437223664811158773730902417043]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=1680895215283185885917511504284592856726] [message="ID:<high:17511504284592856726 low:16808952152831858859 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the periodic resend handling at helper.go:276 and dispatcher memory release at event_collector.go:447. Reproduce the incremental catch-up workload with syncpoints, many dispatchers, and truncate-table DDLs, then verify that dispatcherStatusRequest resends no longer overwhelm memory or lead to OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.