Excessive columns in a table cause reduced CDC decoding speed and increased Changefeed latency (to be optimized independently)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
Description:
During an internal test, significant latency was observed in Changefeed under the following conditions:
Baseline: The test table initially had 52 columns with a row width of approximately 1000 bytes, and the CDC input throughput was stable at 20MB/s;
Operation: 100 consecutive ADD COLUMN "INT NULL DEFAULT NULL COMMENT '测试列'" statements were executed on the table (total columns increased to 152);
Observation: With the increase in the number of columns, the rawKV decoding speed of the Mounter component in the CDC pipeline decreased significantly (see the monitoring chart below), which directly led to elevated Changefeed latency.
| Benchmark Name | Time (ns/op) | Memory (B/op) | Allocations (allocs/op) |
|---|---|---|---|
| BatchDMLEvent_1000000-14 | 1,323,540,125 | 4,848,307,352 | 13,086,256 |
| BatchDMLEvent_1000000_With100IntCols-14 | 6,044,431,166 | 9,097,559,640 | 13,385,793 |
| BatchDMLEvent_100000x10-14 | 1,237,094,792 | 4,636,733,296 | 10,883,477 |
| BatchDMLEvent_100000x10_With100IntCols-14 | 5,483,039,834 | 8,881,322,256 | 11,139,078 |
This issue will be addressed as an independent optimization task.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start with the Mounter component's rawKV decoding path and the BatchDMLEvent_1000000 and BatchDMLEvent_1000000_With100IntCols benchmarks; done means improving decoding speed and reducing Changefeed latency when 100 columns are added.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- performance, stream-processing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100