Reuse Parquet OffsetIndex objects and direct-plan decisions
- 主要语言
- C++
- 星标
- 65
- 派生
- 25
- 平均合并
- 2 天 12 小时
- 30 天内合并 PR
- 80
描述
### Problem
Selective Parquet reads repeat CPU work within one file reader: Arrow 17 reparses OffsetIndex on each lookup, page planning scans page locations for sparse row selections, and decoding rebuilds a direct-read plan just to determine whether compressed row coordinates apply.
The byte cache in #272 reduces repeated I/O but does not eliminate this parsing and planning work.
### Proposed improvement
- Reuse parsed OffsetIndex objects within each retained row-group reader.
- Seek across page gaps for sparse selections and visit each selected page once.
- Reuse the direct-plan decision during decoding.
Preserve index validation, dictionary handling, per-leaf row coordinates and missing-index fallbacks. Keep parsed objects reader-local without introducing a shared data cache or public option.
### Validation
Check selected values and index lifetimes, including sparse/dense selections and the row-group retention limit. Compare against main with the format benchmark, reporting latency and storage bytes without assuming a general speedup.
Implementation: #314.
贡献指南
调研方向
首先检查 issue #314,该 issue 被指定为这项工作的实现。验证 sparse 和 dense 选择中的选定值和索引生命周期,包括 row-group 保留限制,然后比较 format benchmark 与 main 的延迟和存储字节数,不要假设会有普遍的加速效果。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- cpp
- 领域
- data-engineering, performance
- Issue 类型
- 功能
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 25/100