[Feature] Improve scan performance in hot read paths
- 主要語言
- C++
- 星號
- 65
- 分支
- 25
- 平均合併
- 2 天 9 小時
- 30 天內合併 PR
- 82
描述
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.
### Motivation
Recent profiling of highly concurrent scans has revealed several performance bottlenecks caused by
repeated operations on Arrow-returned `shared_ptr` objects in hot read loops.
Two significant cases have been identified:
1. Manifest readers repeatedly call `StructArray::fields()` while processing individual rows. This
copies `shared_ptr` objects and, with GCC 8.3's libstdc++, can introduce substantial lock
contention through `_Sp_locker`, `pthread_mutex_lock`, and futex waits when multiple workers read
manifests concurrently.
2. Avro decoding calls `ArrayBuilder::type()` for every integer and timestamp value. Because this
method returns `std::shared_ptr` by value, concurrent scans repeatedly modify reference
counts on shared Arrow primitive data types, causing cache-line contention. Profiling showed
`ArrayBuilder::type()` and shared-pointer release operations accounting for a large proportion of
samples after the manifest bottleneck was removed.
This issue tracks the broader effort to identify and eliminate similar shared-pointer operations
from scan hot paths. The goal is to cache immutable Arrow metadata at an appropriate batch, reader,
or builder lifetime, while ensuring caches are invalidated whenever the corresponding Arrow object
tree is replaced.
The expected outcome is lower synchronization and reference-counting overhead under concurrent
scans, allowing CPU time to return to actual decoding, memory copying, and buffer management.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
貢獻指南
研究方向
首先,對並行掃描路徑中 manifest reader 的 StructArray::fields() 呼叫和 Avro decoder 的 ArrayBuilder::type() 呼叫進行效能分析。確定適用於不可變 Arrow 中繼資料的 batch、reader 或 builder 生命週期,然後驗證在對應的 Arrow 物件樹被取代時快取會失效。完成的標準是,在並行掃描期間降低 shared-pointer 同步和參考計數的額外負擔。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- cpp
- 領域
- performance
- Issue 類型
- 功能
- 難度
- 5/5
- 預估耗時
- 一週以上
- 活躍度
- 活躍
- 描述清晰度
- 需要釐清
- 新手友好度
- 35/100