apache / apache/paimon-cpp

[Feature] Cut metadata round trips when planning a scan (known manifest sizes, listing probes, parallel manifest-list reads)

已關閉
#337 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
C++
星號
65
分支
25
平均合併
2 天 12 小時
30 天內合併 PR
80

描述

## Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.

## Motivation

Scan planning pays several object-store round trips that are avoidable, either because the answer is already in metadata it has read or because one call can answer what two are asking.

- **Manifest reads re-resolve a length the metadata already carries.** A manifest list records each manifest's `fileSize`, and a snapshot records its base/delta/changelog manifest-list sizes. But `ObjectsFile::Read*` opened these files with a bare `Open(path)`, so on a remote store every manifest and manifest list paid a `getFileStatus`/`HeadObject` round trip just to learn a length planning already had. This is the metadata-path counterpart of the data-file fast path in `Open(const FileStatus&)`.
- **`ListVersionedFileStatus` probes existence before listing.** It called `Exists(dir)` then `ListDir(dir)`. Every file system already lists a missing directory as an empty result rather than an error, so the probe only decided whether to make a call that answers the same question — an extra round trip on every schema/snapshot/versioned-file lookup.
- **Jindo `ListDir` asks the store twice.** It called `Exists(dir)` then `GetFileStatus(dir)`; a single `GetFileStatus` answers both "is it there" and "is it a directory".
- **`ScanMode::ALL` reads the base and delta manifest lists serially.** They are two independent files and neither read depends on the other, so the two metadata round trips are paid one after the other instead of together.

For scans over many manifests, or against a high-latency object store, these round trips are a measurable and entirely avoidable part of planning latency.

## Solution

- Thread an optional known length through the metadata read path: `ObjectsFile::Read`/`ReadIfFileExist`/`ReadArrowBatches`/`ReadFileSegment` gain a `std::optional file_size` (default `std::nullopt`), and a new `OpenForRead` helper opens with `Open(FileStatus(path, size))` when the length is known and falls back to `Open(path)` when it is not. `ManifestFile::ReadBucketEntries` forwards it, `ManifestList::ReadBase/Delta/ChangelogManifests` pass the sizes recorded on the snapshot, and `FileStoreScan` / `SnapshotFileCollector` pass each `ManifestFileMeta::FileSize()`.
- Drop the `Exists()` probe in `FileUtils::ListVersionedFileStatus` and list directly.
- Collapse Jindo `ListDir` to a single `GetFileStatus`, mapping the SDK not-found to an empty listing (as the other file systems do) and propagating any other error.
- In `FileStoreScan::ReadManifestsWithSnapshot`, read the base and delta manifest lists concurrently through the existing `executor_` (`Via` + `CollectAll`), preserving base-then-delta order.

The size fields stay optional, so a snapshot or manifest list written before they existed keeps reading through the `Open(path)` fallback — this is an optimization, not a new requirement on the metadata.

## Anything else?

- The length handed to `Open(FileStatus)` is trusted, not re-validated, so a stale or wrong size would surface as a short or failed read. Manifest and manifest-list files are write-once and never rewritten, and the size is recorded by the same commit that wrote the file, so it cannot go stale underneath the read; a negative length is rejected by the base `Open(const FileStatus&)`.
- The size fast path benefits stores that override `Open(const FileStatus&)` (today `ObjectStoreFileSystem`; `ResolvingFileSystem` forwards it). `JindoFileSystem` currently only overrides `Open(path)`, so for Jindo the manifest-size threading is inert until it gains that override; the Jindo win here is the single-call `ListDir`.
- No storage format or protocol change, and no new public API: the changed signatures are internal `src/paimon` helpers and the new parameter is defaulted.

## Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

貢獻指南

開啟貢獻指南

研究方向

從 ObjectsFile::Read、ReadIfFileExist、ReadArrowBatches 和 ReadFileSegment 開始,接著追蹤 ManifestFile、ManifestList、FileStoreScan 和 SnapshotFileCollector 的呼叫者。也要檢查 FileUtils::ListVersionedFileStatus 和 JindoFileSystem::ListDir。完成的標準是:已知大小時避免 metadata status 呼叫、缺少目錄的列表操作仍能正常運作,且 base/delta manifest 的讀取在並行執行時維持順序。

由索引模型根據 Issue 內容生成。

評估

領域
data-engineering
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
活躍
描述清晰度
基本清楚
新手友好度
58/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。