apache / apache/doris

[Bug] Low throughput of PostgreSQL CDC streaming job, single-core CPU bottleneck on BE write path

Open
#67,123 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

4.1.3

### What's Wrong?

When synchronizing PostgreSQL incremental data via Doris Streaming Job, the import throughput is extremely low and cannot reach business expectations.
Server overall resources are abundant: the machine has 60.7% idle CPU and plenty of free memory, no swap pressure, and IO wait is low. However, the doris_be process consistently occupies 100% single-core CPU, which becomes the strict bottleneck of CDC import.
The upstream Flink task CPU load is only 6.3%, which proves that the data source side is not the bottleneck. The performance bottleneck is completely inside the Doris BE write logic.

Top System Status
%Cpu(s): 24.3 us, 4.7 sy, 0.0 ni, 60.7 id, 6.9 wa, 0.9 hi, 2.5 si, 0.0 st
MiB Mem : 31301.9 total, 12756.9 free, 16703.9 used, 2284.1 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 14598.0 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3923736 root 20 0 31.1g 1.5g 256252 S 102.0 5.0 51:40.35 doris_be
3928325 root 20 0 16.5g 7.6g 19520 S 6.3 24.8 13:12.08 java
- BE single-core full load, multi-core idle, typical single-thread serialization bottleneck
- Upstream Flink Java task almost no pressure
- Memory is sufficient, no IO bottleneck
Image

Perf Flame Graph Analysis
CPU hotspot is concentrated in the single-thread write path of FragmentMgrAsyn asynchronous thread pool:
FragmentMgrAsyn
└── doris::AsyncResultWriter::process_block
└── doris::VTabletWriter::write
└── doris::VRowDistribution::generate_rows_distribution
└── doris::VRowDistribution::__generate_rows_distribution_for_non_auto_partition
├── doris::OlapTableFinder::find_tables
└── doris::Block::dump_data
└── doris::ColumnWithTypeAndName::to_string
└── doris::IDataType::to_string
Two major CPU overheads:
1. Non-auto partition routing: For standard Range partition tables (not AUTO-RANGE), every row needs full tablet metadata lookup via find_tables, which causes heavy per-row computation.
2. Block serialization overhead: Frequent dump_data / to_string conversion consumes massive user CPU during CDC stream load writing.

Image

### What You Expected?

fix it

### How to Reproduce?

To Reproduce
1. Create a standard Range partitioned table (non AUTO-RANGE)
2. Create PostgreSQL CDC streaming job to consume incremental data
3. Generate continuous PG incremental traffic
4. Observe low import throughput and 100% single-core BE CPU usage

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Reproduce the PostgreSQL CDC workload with a standard non-AUTO-RANGE partitioned table and inspect the provided flame-graph path, starting at FragmentMgrAsyn and AsyncResultWriter::process_block. Trace VTabletWriter::write through VRowDistribution::generate_rows_distribution, OlapTableFinder::find_tables, and Block::dump_data. Done means improved import throughput without a persistent single-core BE bottleneck.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
backend, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.