apache / apache/paimon-cpp

[Feature] Improve scan performance in hot read paths

未关闭
#240 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
C++
星标
65
派生
25
平均合并
2 天 12 小时
30 天内合并 PR
80

描述

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.

### Motivation

Recent profiling of highly concurrent scans has revealed several performance bottlenecks caused by
repeated operations on Arrow-returned `shared_ptr` objects in hot read loops.

Two significant cases have been identified:

1. Manifest readers repeatedly call `StructArray::fields()` while processing individual rows. This
copies `shared_ptr` objects and, with GCC 8.3's libstdc++, can introduce substantial lock
contention through `_Sp_locker`, `pthread_mutex_lock`, and futex waits when multiple workers read
manifests concurrently.

2. Avro decoding calls `ArrayBuilder::type()` for every integer and timestamp value. Because this
method returns `std::shared_ptr` by value, concurrent scans repeatedly modify reference
counts on shared Arrow primitive data types, causing cache-line contention. Profiling showed
`ArrayBuilder::type()` and shared-pointer release operations accounting for a large proportion of
samples after the manifest bottleneck was removed.

This issue tracks the broader effort to identify and eliminate similar shared-pointer operations
from scan hot paths. The goal is to cache immutable Arrow metadata at an appropriate batch, reader,
or builder lifetime, while ensuring caches are invalidated whenever the corresponding Arrow object
tree is replaced.

The expected outcome is lower synchronization and reference-counting overhead under concurrent
scans, allowing CPU time to return to actual decoding, memory copying, and buffer management.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

贡献指南

打开贡献指南

调研方向

首先,对并发扫描路径中 manifest reader 的 StructArray::fields() 调用和 Avro decoder 的 ArrayBuilder::type() 调用进行性能分析。确定适用于不可变 Arrow 元数据的 batch、reader 或 builder 生命周期,然后验证在相应的 Arrow 对象树被替换时缓存会失效。完成的标准是,在并发扫描期间降低 shared-pointer 同步和引用计数的开销。

由索引模型根据 Issue 内容生成。

评估

技术栈
cpp
领域
performance
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
活跃
描述清晰度
需要澄清
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。