apache / apache/paimon-cpp

[Feature] Roadmap for Paimon C++ 0.4.0

未關閉
#186 0 則留言 2 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
C++
星號
65
分支
25
平均合併
2 天 9 小時
30 天內合併 PR
82

描述

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.

### Motivation

Looking ahead, Paimon C++ will focus on several areas:

- Storage and retrieval optimizations for vertical workloads, including efficient MAP storage, richer data types such as VECTOR, and complete index support.
- Making newly written data queryable in real time through pluggable real-time writes and memory/disk union reads.
- Following up on important capabilities from the broader Paimon community, including changelog production, Format Table, and extensions to Data Evolution.

### Solution

#### 1. Storage and retrieval optimizations for vertical workloads

- [x] Support shared-shredding columnar storage for `MAP` based on [PIP-43](https://cwiki.apache.org/confluence/spaces/PAIMON/pages/430408347/PIP-43+Columnar+Storage+Optimization+for+MAP+Type+in+Paimon), including adaptive physical columns, field mappings, overflow storage, and end-to-end reads and writes.
- [ ] Support richer data types, starting with `VECTOR` based on [PIP-40](https://cwiki.apache.org/confluence/spaces/PAIMON/pages/399279132/PIP-40+Introduce+a+new+Vector+data+type), including schema representation, storage, reads, writes, and Data Evolution.
- [ ] Complete File Index support by adding index generation to the existing read path.
- [ ] Continue improving storage layout, predicate pushdown, column pruning, point lookups, and index-assisted retrieval for workload-specific scenarios.

#### 2. Making real-time data queryable

- [ ] Support pluggable real-time writes and memory/disk union reads based on [PIP-46](https://cwiki.apache.org/confluence/spaces/PAIMON/pages/444334302/PIP-46+Support+pluggable+real-time+writes+and+memory+disk+union+reads+for+Paimon+C), tracked by [#158](https://github.com/apache/paimon-cpp/issues/158).
- [ ] Make data held by an active writer queryable before it is committed into a snapshot.
- [ ] Guarantee consistent query results without missing or duplicate rows while writing, committing, and reclaiming memory segments concurrently.
- [ ] Support append tables first, followed by primary-key tables, deletion-vector mode, and additional production scenarios.

#### 3. Following up on Paimon community capabilities

- [ ] Support changelog production, including `input`, `lookup`, and `full-compaction` modes, tracked by [#174](https://github.com/apache/paimon-cpp/issues/174).
- [ ] Support Format Table reads and writes for Hive-style file directories, tracked by [#170](https://github.com/apache/paimon-cpp/issues/170).
- [ ] Extend Data Evolution to work with deletion vectors, tracked by [#169](https://github.com/apache/paimon-cpp/issues/169).
- [ ] Extend Data Evolution to primary-key tables and support compaction across evolved field groups, tracked by [#204](https://github.com/apache/paimon-cpp/issues/204).
- [ ] Continue following relevant Paimon features and make them available through native Paimon C++ APIs where appropriate.

Each roadmap item should be delivered through focused issues and reviewable pull requests, with unit tests, integration tests, documentation, and compatibility validation where applicable.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

貢獻指南

開啟貢獻指南

研究方向

從連結的聚焦 issue 開始,尤其是關於即時寫入和 union 讀取的 #158、關於 changelog 產生的 #174、關於 Format Table 的 #170,以及關於 Data Evolution 的 #169 或 #204。在適用的情況下閱讀所參照的 PIP 文件,並選擇一個範圍明確的 roadmap 項目。完成的要求是進行一項聚焦且可供 review 的變更,並包含相關的 unit 和 integration 測試、文件以及相容性驗證。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
cpp
領域
databases
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
冷清
描述清晰度
需要釐清
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。