apache / apache/iceberg-python
Remove deleted data files with expire_snapshots
- 主要语言
- Python
- 星标
- 1.1k
- 派生
- 581
- 平均合并
- 1 天 17 小时
- 30 天内合并 PR
- 78
描述
### Feature Request / Improvement
Running an expire snapshots operation will only rewrite the metadata file without the expired snapshots (and refs/ statistics). It does not delete deleted data files referenced only by the expired snapshots. This can be observed by deleting an entire table and calling `expire_snapshots` - the data files still exist. Trino and spark both clean up deleted data files when all snapshots referencing them are expired.
From the spec:
```
When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID
in which the file was deleted and status 2 (deleted). The file may be deleted from the file system
when the snapshot in which it was deleted is garbage collected, assuming that older snapshots
have also been garbage collected [1].
...
1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is
garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It
is easier to track what files are deleted in a snapshot and delete them when that snapshot expires.
It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge
cases where incremental deletes can break table snapshots.
```
Happy to work on this if others agree that it should be added :)
贡献指南
这个仓库没有索引到贡献指南
调研方向
首先跟踪 expire_snapshots 操作以及它如何重写元数据,然后检查对标记为已删除的 manifest 条目的处理。根据所描述的整表删除场景和相关的现有测试确认其行为;完成的标准是,只有过期快照引用的数据文件会被删除,而保留快照仍需要的文件会继续保留。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- databases
- Issue 类型
- 功能
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 45/100