CDC OOM when creating 300 changefeed in a short time.
Open
Nobody has claimed this yet.
severity/moderate
type/bug
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
There are 10M tables upstream.
The creation of each changefeed requires loading all the tables from the upstream. This means that the creation of each changefeed takes 20 minutes to load the tables and consumes approximately 30GB of memory. Which casue OOM when creating 300 changefeeds in a short time.
CDC Version
> ./cdc version
Release Version: v9.0.0-alpha-185-gb3c5be2a
Git Commit Hash: b3c5be2a5565c111a4babfe10b5b7f3bfb87da74
Git Branch: master
UTC Build Time: 2025-03-27 03:10:59
Go Version: go version go1.23.4 darwin/arm64
Failpoint Build: false
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start from the CDC changefeed creation flow described in the report and trace how loading all upstream tables is performed for each changefeed. Reproduce the short-interval creation scenario with a large table set, then measure memory use and verify that creating many changefeeds no longer causes OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100