apache / apache/paimon-cpp

[Feature] Prefetch the late-materialization payload ranges instead of reading them on demand

Fermée
#341 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
C++
Étoiles
65
Forks
25
Merge moyen
2 j 12 h
PR mergées (30 j)
80

Description

## Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.

## Motivation

Late materialization reads a data file in two passes: a probe pass over the predicate fields, then a payload pass over the remaining fields for the matched rows only. The shared read-ahead cache is fed once per read-range generation through `PrefetchFileBatchReader::PreBufferRange()`, before any read starts. At that point the payload pass cannot know which pages hold the matched rows — that depends on the probe result — so `PreBufferRange()` only reports the probe ranges (there was an explicit TODO for exactly this). The payload pass is therefore never prefetched: every payload read misses the cache and waits for its own underlying IO, serialized against the decode, on the pass that touches the wide columns.

## Solution

Let a reader report byte ranges that only become known after reading has started, and let the shared cache register them mid-read.

- New `ReadAheadCache::AddRanges(ranges, expected_round)` registers ranges into an already-initialized cache and is safe to call repeatedly and concurrently with `Read()`. It merges the new ranges into the disjoint, offset-ordered pending list, registering only the parts no registered range covers and dropping the overlap (the round that registered it is already fetching those bytes), then rebuilds the per-range cached flags so an already-fetched range is not fetched twice. The registered part is cut at a new `CacheConfig` knob `late_range_size_limit` (default 8 MiB, smaller than the 32 MiB `range_size_limit`) so a large pass is fetched by several concurrent requests rather than one long one; a new `Warmup(from_offset)` starts fetching from the first newly-registered range instead of from the head.
- A registration round bounds the lifetime: every `Init()` opens a round identified by `RegistrationRound()`, and `AddRanges()` drops everything when `expected_round` is not the open round, so a pass that outlived its generation — the cache was reset for a new read-range generation, or released by `Close()` — registers nothing instead of prefetching bytes nobody reads. The round counter is monotonic across `Reset()` so a stale round is never mistaken for a new one.
- New `PrefetchFileBatchReader::PreBufferSink` and `SetPreBufferSink()`: `PrefetchFileBatchReaderImpl` installs a sink on each sub-reader that tags the reported ranges with the current round, calls `AddRanges`, and warms up from the first new range. `LateMaterializingFileBatchReader` reports the payload ranges through the sink once the probe pass has refined the inner reader's target pages, and surfaces a failure to compute them (they come from the file metadata) rather than swallowing it.
- New metrics `read-ahead-cache.late.registered` / `.registered-bytes` / `.dropped` / `.dropped-bytes`, counted after coalescing and splitting, so `registered-bytes` and `dropped-bytes` together account for every reported byte.

## Anything else?

Adds public API under `include/paimon/`: `PrefetchFileBatchReader::PreBufferSink` / `SetPreBufferSink()` and `CacheConfig::GetLateRangeSizeLimit()` / `SetLateRangeSizeLimit()`. `ReadAheadCache::AddRanges` / `RegistrationRound` / `Warmup(offset)` and the new counter names live in the internal header. No storage format or protocol change.

## Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par suivre PrefetchFileBatchReader::PreBufferRange(), ReadAheadCache::Init(), Read(), Reset() et Close(), puis examinez les APIs publiques sous include/paimon/. Suivez la manière dont LateMaterializingFileBatchReader détermine les plages de payload et dont les métriques de cache existantes sont enregistrées. Le travail est terminé lorsque les plages tardives sont enregistrées et préchauffées de manière sûre entre les différentes passes, que les enregistrements obsolètes sont supprimés, que les règles de concurrence et de chevauchement sont respectées et que les nouvelles métriques comptabilisent les octets enregistrés et supprimés.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
cpp
Domaine
data-engineering, performance
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
Active
Clarté
Clairement spécifiée
Accessibilité débutants
38/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.