CDC OOM When There Are 60k Dispatchers with Syncpoint Enabled and Many Truncate Table DDLs
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
During incremental catch-up, due to the need to process syncpoints/DDLs accumulated over a period of time, a large number of dispatcherStatusRequest are repeatedly sent for processing.
Then, because there are a relatively large number of dispatchers, the maintainer's handling speed cannot keep up with their resending speed. Eventually, this overwhelms the system, resulting in an OOM situation.
➜ log grep "resend task periodic resend" cdc-2026-04-15T18-54-13.836.log | wc -l
322481
➜ log tail cdc-2026-04-15T18-54-13.836.log
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=99651912610150768516142891601317547648] [message="ID:<high:6142891601317547648 low:9965191261015076851 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=56387882334628808071329043824189030809] [message="ID:<high:1329043824189030809 low:5638788233462880807 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]
[2026/04/15 18:54:13.836 +08:00] [INFO] [event_collector.go:447] ["release dispatcher memory in DS"] [dispatcherID=137131577023193707788927581790319772544]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=1659228167089718475513856676581264965551] [message="ID:<high:13856676581264965551 low:16592281670897184755 > state:<IsBlocked:true BlockTs:465633269514240000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=10]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=177467660874281160245570627732352186044] [message="ID:<high:5570627732352186044 low:17746766087428116024 > state:<IsBlocked:true BlockTs:465633269514240000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=10]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=35515166653188255502549568495179157148] [message="ID:<high:2549568495179157148 low:3551516665318825550 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]
[2026/04/15 18:54:13.836 +08:00] [INFO] [event_collector.go:447] ["release dispatcher memory in DS"] [dispatcherID=54953054923390040866848125201506941597]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=131778944883591698812614583334147500163] [message="ID:<high:12614583334147500163 low:1317789448835916988 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]
[2026/04/15 18:54:13.836 +08:00] [INFO] [event_collector.go:447] ["release dispatcher memory in DS"] [dispatcherID=470363268437223664811158773730902417043]
[2026/04/15 18:54:13.836 +08:00] [INFO] [helper.go:276] ["resend task periodic resend"] [dispatcherID=1680895215283185885917511504284592856726] [message="ID:<high:17511504284592856726 low:16808952152831858859 > state:<IsBlocked:true BlockTs:465633206599680000 BlockTables:<> IsSyncPoint:true stage:WAITING > "] [executeCount=20]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the periodic resend handling at helper.go:276 and dispatcher memory release at event_collector.go:447. Reproduce the incremental catch-up workload with syncpoints, many dispatchers, and truncate-table DDLs, then verify that dispatcherStatusRequest resends no longer overwhelm memory or lead to OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100