apache / apache/paimon-cpp

[Feature] Maintain source-backed primary-key BTree indexes during compaction

未關閉
#291 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
C++
星號
65
分支
25
平均合併
2 天 9 小時
30 天內合併 PR
82

描述

### Search before asking

- [x] I searched in the issues and found nothing similar.

### Motivation

#192 and #194 added the source-backed primary-key BTree read path. Paimon C++ writers still need the corresponding maintenance path: after compaction changes the active source files of a data level, a missing or stale payload leaves that level uncovered and queries fall back to normal file scans.

Paimon C++ should maintain these payloads during fixed-bucket primary-key writes and compaction, using the existing Java-compatible source metadata, BTree payload format, and index manifests.

### Solution

Add the source-backed primary-key BTree maintenance lifecycle for fixed-bucket primary-key tables:

1. Validate the Java-equivalent table and index prerequisites.
2. Restore committed source-backed payload metadata into bucket writers without mixing Data Evolution payloads.
3. Build one payload per indexed field and positive data level from physical source rows, then commit matching index additions and deletions in the same snapshot as the data changes.
4. Reconcile missing, stale, duplicate, replaced, removed-definition, and empty-level payloads during compaction.
5. Isolate build failures to the affected field and level so reads safely fall back to normal scans and a later maintenance attempt can rebuild the payload.
6. Retain live index files during snapshot expiration and orphan cleanup, including tag and branch safety and external-file deletion retries.

Reuse the existing storage formats and internal reader, writer, sort-buffer, path, manifest, and commit abstractions. Do not introduce a new index family or storage protocol.

The implementation is in #245. It keeps maintenance synchronous; Java asynchronous scheduling, manual rebuild actions, realtime writers, and postpone-bucket writers remain outside this scope.

### Anything else?

This is a maintenance-path follow-up to the read-path work in #192 and #194. It ports an existing Java capability, so no separate PIP is proposed.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

貢獻指南

開啟貢獻指南

研究方向

先閱讀 #245 中參照的實作,以及 #192 和 #194 中由原始碼支援的 primary-key read path。追蹤現有的儲存格式,以及內部 reader 和 writer、manifest、path、sort-buffer 與 commit 抽象。完成的標準是 fixed-bucket writes 和 compaction 能維持按欄位劃分且位於正層級的 payload,同時 cleanup 和失敗的 builds 保留安全的 scan fallback。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
cpp
領域
databases
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
停滯
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。