apache / apache/paimon-cpp

Reuse Parquet OffsetIndex objects and direct-plan decisions

未关闭
#328 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C++
星标
65
派生
25
平均合并
2 天 12 小时
30 天内合并 PR
80

描述

### Problem

Selective Parquet reads repeat CPU work within one file reader: Arrow 17 reparses OffsetIndex on each lookup, page planning scans page locations for sparse row selections, and decoding rebuilds a direct-read plan just to determine whether compressed row coordinates apply.

The byte cache in #272 reduces repeated I/O but does not eliminate this parsing and planning work.

### Proposed improvement

- Reuse parsed OffsetIndex objects within each retained row-group reader.
- Seek across page gaps for sparse selections and visit each selected page once.
- Reuse the direct-plan decision during decoding.

Preserve index validation, dictionary handling, per-leaf row coordinates and missing-index fallbacks. Keep parsed objects reader-local without introducing a shared data cache or public option.

### Validation

Check selected values and index lifetimes, including sparse/dense selections and the row-group retention limit. Compare against main with the format benchmark, reporting latency and storage bytes without assuming a general speedup.

Implementation: #314.

贡献指南

打开贡献指南

调研方向

首先检查 issue #314,该 issue 被指定为这项工作的实现。验证 sparse 和 dense 选择中的选定值和索引生命周期,包括 row-group 保留限制,然后比较 format benchmark 与 main 的延迟和存储字节数,不要假设会有普遍的加速效果。

由索引模型根据 Issue 内容生成。

评估

技术栈
cpp
领域
data-engineering, performance
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。