apache / apache/paimon-cpp

[Feature] Prefetch the late-materialization payload ranges instead of reading them on demand

已關閉
#341 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
C++
星號
65
分支
25
平均合併
2 天 9 小時
30 天內合併 PR
82

描述

## Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.

## Motivation

Late materialization reads a data file in two passes: a probe pass over the predicate fields, then a payload pass over the remaining fields for the matched rows only. The shared read-ahead cache is fed once per read-range generation through `PrefetchFileBatchReader::PreBufferRange()`, before any read starts. At that point the payload pass cannot know which pages hold the matched rows — that depends on the probe result — so `PreBufferRange()` only reports the probe ranges (there was an explicit TODO for exactly this). The payload pass is therefore never prefetched: every payload read misses the cache and waits for its own underlying IO, serialized against the decode, on the pass that touches the wide columns.

## Solution

Let a reader report byte ranges that only become known after reading has started, and let the shared cache register them mid-read.

- New `ReadAheadCache::AddRanges(ranges, expected_round)` registers ranges into an already-initialized cache and is safe to call repeatedly and concurrently with `Read()`. It merges the new ranges into the disjoint, offset-ordered pending list, registering only the parts no registered range covers and dropping the overlap (the round that registered it is already fetching those bytes), then rebuilds the per-range cached flags so an already-fetched range is not fetched twice. The registered part is cut at a new `CacheConfig` knob `late_range_size_limit` (default 8 MiB, smaller than the 32 MiB `range_size_limit`) so a large pass is fetched by several concurrent requests rather than one long one; a new `Warmup(from_offset)` starts fetching from the first newly-registered range instead of from the head.
- A registration round bounds the lifetime: every `Init()` opens a round identified by `RegistrationRound()`, and `AddRanges()` drops everything when `expected_round` is not the open round, so a pass that outlived its generation — the cache was reset for a new read-range generation, or released by `Close()` — registers nothing instead of prefetching bytes nobody reads. The round counter is monotonic across `Reset()` so a stale round is never mistaken for a new one.
- New `PrefetchFileBatchReader::PreBufferSink` and `SetPreBufferSink()`: `PrefetchFileBatchReaderImpl` installs a sink on each sub-reader that tags the reported ranges with the current round, calls `AddRanges`, and warms up from the first new range. `LateMaterializingFileBatchReader` reports the payload ranges through the sink once the probe pass has refined the inner reader's target pages, and surfaces a failure to compute them (they come from the file metadata) rather than swallowing it.
- New metrics `read-ahead-cache.late.registered` / `.registered-bytes` / `.dropped` / `.dropped-bytes`, counted after coalescing and splitting, so `registered-bytes` and `dropped-bytes` together account for every reported byte.

## Anything else?

Adds public API under `include/paimon/`: `PrefetchFileBatchReader::PreBufferSink` / `SetPreBufferSink()` and `CacheConfig::GetLateRangeSizeLimit()` / `SetLateRangeSizeLimit()`. `ReadAheadCache::AddRanges` / `RegistrationRound` / `Warmup(offset)` and the new counter names live in the internal header. No storage format or protocol change.

## Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

貢獻指南

開啟貢獻指南

研究方向

先追蹤 PrefetchFileBatchReader::PreBufferRange()、ReadAheadCache::Init()、Read()、Reset() 和 Close(),接著檢查 include/paimon/ 下的公開 API。了解 LateMaterializingFileBatchReader 如何判定 payload 範圍,以及現有快取指標如何記錄。完成的標準是:late 範圍能跨回合安全地註冊並預熱、過時的註冊會被捨棄、並行與重疊規則能獲得遵守,而且新的指標能計算已註冊與已捨棄的位元組數。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
cpp
領域
data-engineering, performance
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
活躍
描述清晰度
描述清楚
新手友好度
38/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。