apache / apache/iceberg-python
_DeleteFiles.delete_data_file() silently drops explicit file references
- 主要语言
- Python
- 星标
- 1.1k
- 派生
- 581
- 平均合并
- 1 天 17 小时
- 30 天内合并 PR
- 78
描述
## Description
`_DeleteFiles.delete_data_file()` silently drops explicit file references because `_compute_deletes` resets `self._deleted_data_files = set()` before scanning manifests by predicate.
The `delete_data_file()` method is inherited from `_SnapshotProducer` and adds to `self._deleted_data_files`. However, when `_DeleteFiles._compute_deletes` runs (triggered by `_deleted_entries()`), it resets this field to an empty set and repopulates it only from files matched by the predicate evaluator. Any explicitly added files are silently lost.
While the high-level `table.delete()` API only uses predicates (so this isn't hit in normal usage), the low-level `update_snapshot().delete()` API exposes `delete_data_file()` as a public method. Calling it produces no error and no effect.
## Reproduction
```python
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField
catalog = load_catalog("default")
catalog.create_namespace("default")
table = catalog.create_table(
"default.delete_explicit",
Schema(NestedField(1, "x", LongType(), required=False)),
)
table.append(pa.table({"x": [1, 2, 3]}))
data_file = next(iter(table.scan().plan_files())).file
# This should delete the file but silently does nothing
with table.transaction() as tx:
delete_snapshot = tx.update_snapshot().delete()
delete_snapshot.delete_data_file(data_file)
# Bug: table still has [1, 2, 3]
print(table.scan().to_arrow()["x"].to_pylist())
```
Expected: table is empty after commit.
Actual: table still has `[1, 2, 3]`.
## Root cause
In `pyiceberg/table/update/snapshot.py`, `_DeleteFiles._compute_deletes` (line ~598):
```python
self._deleted_data_files = set() # <-- overwrites any explicit files added via delete_data_file()
```
Then it repopulates from predicate matches only. Since no predicate was set (`AlwaysFalse` by default), nothing matches, nothing is deleted.
## Suggested fix
Before resetting, preserve explicit files and include them in the deletion scan:
```python
# Preserve files explicitly requested for deletion
explicit_deletes = set(self._deleted_data_files)
self._deleted_data_files = set()
# ... existing predicate-based scan ...
# After the scan, also mark explicitly requested files as deleted:
for entry in manifest_entries:
if entry.data_file in explicit_deletes:
# mark as deleted
```
Alternatively, prevent calling `delete_data_file()` on `_DeleteFiles` entirely by raising `NotImplementedError`.
## Related
Follow-up observation from #3818 review. The `_OverwriteFiles` path handles `delete_data_file()` correctly because it does not reset the field.
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 pyiceberg/table/update/snapshot.py 中 _DeleteFiles._compute_deletes 的大约第 598 行开始,跟踪 delete_data_file() 如何记录显式指定的文件。运行 issue 中的复现,然后验证显式引用的数据文件会在 commit 后被删除,同时基于 predicate 的删除仍然有效。当表为空而不是保留 [1, 2, 3] 时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- databases
- Issue 类型
- 缺陷
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 活跃度
- 活跃
- 描述清晰度
- 描述清楚
- 新手友好度
- 72/100