apache / apache/paimon-cpp

Reuse Parquet OffsetIndex objects and direct-plan decisions

未關閉
#328 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
C++
星號
65
分支
25
平均合併
2 天 9 小時
30 天內合併 PR
82

描述

### Problem

Selective Parquet reads repeat CPU work within one file reader: Arrow 17 reparses OffsetIndex on each lookup, page planning scans page locations for sparse row selections, and decoding rebuilds a direct-read plan just to determine whether compressed row coordinates apply.

The byte cache in #272 reduces repeated I/O but does not eliminate this parsing and planning work.

### Proposed improvement

- Reuse parsed OffsetIndex objects within each retained row-group reader.
- Seek across page gaps for sparse selections and visit each selected page once.
- Reuse the direct-plan decision during decoding.

Preserve index validation, dictionary handling, per-leaf row coordinates and missing-index fallbacks. Keep parsed objects reader-local without introducing a shared data cache or public option.

### Validation

Check selected values and index lifetimes, including sparse/dense selections and the row-group retention limit. Compare against main with the format benchmark, reporting latency and storage bytes without assuming a general speedup.

Implementation: #314.

貢獻指南

開啟貢獻指南

研究方向

首先檢查 issue #314,該 issue 被指定為這項工作的實作。驗證 sparse 和 dense 選取中的選定值和索引生命週期,包括 row-group 保留限制,然後比較 format benchmark 與 main 的延遲和儲存位元組數,不要假設會有普遍的加速效果。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
cpp
領域
data-engineering, performance
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。