Track Iceberg v3 Feature Support (Roadmap for 0.4.0)
- 主要语言
- C++
- 星标
- 221
- 派生
- 124
- 平均合并
- 1 天 16 小时
- 30 天内合并 PR
- 21
描述
We already have **partial support for Iceberg v3** in the codebase (e.g., Row Lineage and partition statistics). However, a number of key v3 features are still missing.
This issue serves as a **parent tracking issue** for the remaining v3 work we plan to (partially) include in the **0.3.0 release**.
If you are interested in working on any of the items below, please **open a dedicated sub-issue** and link it back to this one.
## V3 related
### 1. Schema-Level Support for New v3 Types
Add support for all newly introduced v3 types at the schema level, including:
- [x] `timestamp_ns`, `timestamptz_ns` @zhjwpku
- [x] `unknown` @manuzhang
- [ ] `variant` @zhjwpku
- [x] `geometry`, `geography` @wgtmac
### 2. Column Default Values (Schema Evolution)
Enable schema evolution with default values:
- [x] Support `initial-default` and `write-default` @huan233usc
- [ ] Apply defaults correctly during read and write paths
### ~~3. Multi-Argument Transforms (Not planned, as they are not supported by the Java implementation)~~
### 4. Binary Deletion Vectors
Add support for deletion vectors (DV):
- [x] Puffin-based DV parsing @zhaoxuan1994
### 5. Table-Level Encryption Metadata (Not planned for 0.4.0) @shangxinli
Support encryption-related metadata defined in v3:
- [ ] Parse and preserve encryption metadata
### 6. Puffin-Based Statistics (Not planned for 0.4.0)
Support table and partition statistics stored in Puffin files:
- [ ] Metadata parsing
- [ ] Optional integration into scan planning
## Table update and maintenance: #364
- [x] OverwriteFiles @lishuxu
- [x] RowDelta @manuzhang
- [x] MergeAppend @zhjwpku
- [x] DeleteFiles @manuzhang
- [x] RewriteFiles @WZhuo
- [ ] ReplacePartitions @shangxinli
- [ ] RewriteManifests @zhjwpku
- [ ] CherryPickOperation @shangxinli
## Miscellaneous
- [ ] Metadata table inspection @All-less @WZhuo
- [x] LRU cache @HuaHuaY
- [ ] Logging interface and spdlog integration @kamcheungting-db
If anything is missing or you have suggestions, feel free to comment or open additional sub-issues.
贡献指南
这个仓库没有索引到贡献指南
调研方向
这是一个父级跟踪 issue,而不是一个自包含的任务,其中没有指定文件、测试或入口点。选择一个尚未勾选的项目,例如 schema 默认值、加密元数据或表更新操作,然后打开一个与此处关联的专用子 issue。完成意味着完成该范围明确的项目,并更新其对应的检查清单条目。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- cpp
- 领域
- data-engineering
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 冷清
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100