[Feature] Prefetch the late-materialization payload ranges instead of reading them on demand
- 主要語言
- C++
- 星號
- 65
- 分支
- 25
- 平均合併
- 2 天 9 小時
- 30 天內合併 PR
- 82
描述
## Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.
## Motivation
Late materialization reads a data file in two passes: a probe pass over the predicate fields, then a payload pass over the remaining fields for the matched rows only. The shared read-ahead cache is fed once per read-range generation through `PrefetchFileBatchReader::PreBufferRange()`, before any read starts. At that point the payload pass cannot know which pages hold the matched rows — that depends on the probe result — so `PreBufferRange()` only reports the probe ranges (there was an explicit TODO for exactly this). The payload pass is therefore never prefetched: every payload read misses the cache and waits for its own underlying IO, serialized against the decode, on the pass that touches the wide columns.
## Solution
Let a reader report byte ranges that only become known after reading has started, and let the shared cache register them mid-read.
- New `ReadAheadCache::AddRanges(ranges, expected_round)` registers ranges into an already-initialized cache and is safe to call repeatedly and concurrently with `Read()`. It merges the new ranges into the disjoint, offset-ordered pending list, registering only the parts no registered range covers and dropping the overlap (the round that registered it is already fetching those bytes), then rebuilds the per-range cached flags so an already-fetched range is not fetched twice. The registered part is cut at a new `CacheConfig` knob `late_range_size_limit` (default 8 MiB, smaller than the 32 MiB `range_size_limit`) so a large pass is fetched by several concurrent requests rather than one long one; a new `Warmup(from_offset)` starts fetching from the first newly-registered range instead of from the head.
- A registration round bounds the lifetime: every `Init()` opens a round identified by `RegistrationRound()`, and `AddRanges()` drops everything when `expected_round` is not the open round, so a pass that outlived its generation — the cache was reset for a new read-range generation, or released by `Close()` — registers nothing instead of prefetching bytes nobody reads. The round counter is monotonic across `Reset()` so a stale round is never mistaken for a new one.
- New `PrefetchFileBatchReader::PreBufferSink` and `SetPreBufferSink()`: `PrefetchFileBatchReaderImpl` installs a sink on each sub-reader that tags the reported ranges with the current round, calls `AddRanges`, and warms up from the first new range. `LateMaterializingFileBatchReader` reports the payload ranges through the sink once the probe pass has refined the inner reader's target pages, and surfaces a failure to compute them (they come from the file metadata) rather than swallowing it.
- New metrics `read-ahead-cache.late.registered` / `.registered-bytes` / `.dropped` / `.dropped-bytes`, counted after coalescing and splitting, so `registered-bytes` and `dropped-bytes` together account for every reported byte.
## Anything else?
Adds public API under `include/paimon/`: `PrefetchFileBatchReader::PreBufferSink` / `SetPreBufferSink()` and `CacheConfig::GetLateRangeSizeLimit()` / `SetLateRangeSizeLimit()`. `ReadAheadCache::AddRanges` / `RegistrationRound` / `Warmup(offset)` and the new counter names live in the internal header. No storage format or protocol change.
## Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
貢獻指南
研究方向
先追蹤 PrefetchFileBatchReader::PreBufferRange()、ReadAheadCache::Init()、Read()、Reset() 和 Close(),接著檢查 include/paimon/ 下的公開 API。了解 LateMaterializingFileBatchReader 如何判定 payload 範圍,以及現有快取指標如何記錄。完成的標準是:late 範圍能跨回合安全地註冊並預熱、過時的註冊會被捨棄、並行與重疊規則能獲得遵守,而且新的指標能計算已註冊與已捨棄的位元組數。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- cpp
- 領域
- data-engineering, performance
- Issue 類型
- 功能
- 難度
- 5/5
- 預估耗時
- 一週以上
- 活躍度
- 活躍
- 描述清晰度
- 描述清楚
- 新手友好度
- 38/100