[Bug] Low throughput of PostgreSQL CDC streaming job, single-core CPU bottleneck on BE write path
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
4.1.3
### What's Wrong?
When synchronizing PostgreSQL incremental data via Doris Streaming Job, the import throughput is extremely low and cannot reach business expectations.
Server overall resources are abundant: the machine has 60.7% idle CPU and plenty of free memory, no swap pressure, and IO wait is low. However, the doris_be process consistently occupies 100% single-core CPU, which becomes the strict bottleneck of CDC import.
The upstream Flink task CPU load is only 6.3%, which proves that the data source side is not the bottleneck. The performance bottleneck is completely inside the Doris BE write logic.
Top System Status
%Cpu(s): 24.3 us, 4.7 sy, 0.0 ni, 60.7 id, 6.9 wa, 0.9 hi, 2.5 si, 0.0 st
MiB Mem : 31301.9 total, 12756.9 free, 16703.9 used, 2284.1 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 14598.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3923736 root 20 0 31.1g 1.5g 256252 S 102.0 5.0 51:40.35 doris_be
3928325 root 20 0 16.5g 7.6g 19520 S 6.3 24.8 13:12.08 java
- BE single-core full load, multi-core idle, typical single-thread serialization bottleneck
- Upstream Flink Java task almost no pressure
- Memory is sufficient, no IO bottleneck
Perf Flame Graph Analysis
CPU hotspot is concentrated in the single-thread write path of FragmentMgrAsyn asynchronous thread pool:
FragmentMgrAsyn
└── doris::AsyncResultWriter::process_block
└── doris::VTabletWriter::write
└── doris::VRowDistribution::generate_rows_distribution
└── doris::VRowDistribution::__generate_rows_distribution_for_non_auto_partition
├── doris::OlapTableFinder::find_tables
└── doris::Block::dump_data
└── doris::ColumnWithTypeAndName::to_string
└── doris::IDataType::to_string
Two major CPU overheads:
1. Non-auto partition routing: For standard Range partition tables (not AUTO-RANGE), every row needs full tablet metadata lookup via find_tables, which causes heavy per-row computation.
2. Block serialization overhead: Frequent dump_data / to_string conversion consumes massive user CPU during CDC stream load writing.
### What You Expected?
fix it
### How to Reproduce?
To Reproduce
1. Create a standard Range partitioned table (non AUTO-RANGE)
2. Create PostgreSQL CDC streaming job to consume incremental data
3. Generate continuous PG incremental traffic
4. Observe low import throughput and 100% single-core BE CPU usage
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Reproduce the PostgreSQL CDC workload with a standard non-AUTO-RANGE partitioned table and inspect the provided flame-graph path, starting at FragmentMgrAsyn and AsyncResultWriter::process_block. Trace VTabletWriter::write through VRowDistribution::generate_rows_distribution, OlapTableFinder::find_tables, and Block::dump_data. Done means improved import throughput without a persistent single-core BE bottleneck.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100