Reuse Parquet OffsetIndex objects and direct-plan decisions
- 主要語言
- C++
- 星號
- 65
- 分支
- 25
- 平均合併
- 2 天 9 小時
- 30 天內合併 PR
- 82
描述
### Problem
Selective Parquet reads repeat CPU work within one file reader: Arrow 17 reparses OffsetIndex on each lookup, page planning scans page locations for sparse row selections, and decoding rebuilds a direct-read plan just to determine whether compressed row coordinates apply.
The byte cache in #272 reduces repeated I/O but does not eliminate this parsing and planning work.
### Proposed improvement
- Reuse parsed OffsetIndex objects within each retained row-group reader.
- Seek across page gaps for sparse selections and visit each selected page once.
- Reuse the direct-plan decision during decoding.
Preserve index validation, dictionary handling, per-leaf row coordinates and missing-index fallbacks. Keep parsed objects reader-local without introducing a shared data cache or public option.
### Validation
Check selected values and index lifetimes, including sparse/dense selections and the row-group retention limit. Compare against main with the format benchmark, reporting latency and storage bytes without assuming a general speedup.
Implementation: #314.
貢獻指南
研究方向
首先檢查 issue #314,該 issue 被指定為這項工作的實作。驗證 sparse 和 dense 選取中的選定值和索引生命週期,包括 row-group 保留限制,然後比較 format benchmark 與 main 的延遲和儲存位元組數,不要假設會有普遍的加速效果。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- cpp
- 領域
- data-engineering, performance
- Issue 類型
- 功能
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 25/100