apache / apache/iceberg-python

_DeleteFiles.delete_data_file() silently drops explicit file references

未关闭
#3,857 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

## Description

`_DeleteFiles.delete_data_file()` silently drops explicit file references because `_compute_deletes` resets `self._deleted_data_files = set()` before scanning manifests by predicate.

The `delete_data_file()` method is inherited from `_SnapshotProducer` and adds to `self._deleted_data_files`. However, when `_DeleteFiles._compute_deletes` runs (triggered by `_deleted_entries()`), it resets this field to an empty set and repopulates it only from files matched by the predicate evaluator. Any explicitly added files are silently lost.

While the high-level `table.delete()` API only uses predicates (so this isn't hit in normal usage), the low-level `update_snapshot().delete()` API exposes `delete_data_file()` as a public method. Calling it produces no error and no effect.

## Reproduction

```python
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField

catalog = load_catalog("default")
catalog.create_namespace("default")
table = catalog.create_table(
"default.delete_explicit",
Schema(NestedField(1, "x", LongType(), required=False)),
)
table.append(pa.table({"x": [1, 2, 3]}))

data_file = next(iter(table.scan().plan_files())).file

# This should delete the file but silently does nothing
with table.transaction() as tx:
delete_snapshot = tx.update_snapshot().delete()
delete_snapshot.delete_data_file(data_file)

# Bug: table still has [1, 2, 3]
print(table.scan().to_arrow()["x"].to_pylist())
```

Expected: table is empty after commit.
Actual: table still has `[1, 2, 3]`.

## Root cause

In `pyiceberg/table/update/snapshot.py`, `_DeleteFiles._compute_deletes` (line ~598):

```python
self._deleted_data_files = set() # <-- overwrites any explicit files added via delete_data_file()
```

Then it repopulates from predicate matches only. Since no predicate was set (`AlwaysFalse` by default), nothing matches, nothing is deleted.

## Suggested fix

Before resetting, preserve explicit files and include them in the deletion scan:

```python
# Preserve files explicitly requested for deletion
explicit_deletes = set(self._deleted_data_files)
self._deleted_data_files = set()

# ... existing predicate-based scan ...
# After the scan, also mark explicitly requested files as deleted:
for entry in manifest_entries:
if entry.data_file in explicit_deletes:
# mark as deleted
```

Alternatively, prevent calling `delete_data_file()` on `_DeleteFiles` entirely by raising `NotImplementedError`.

## Related

Follow-up observation from #3818 review. The `_OverwriteFiles` path handles `delete_data_file()` correctly because it does not reset the field.

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 pyiceberg/table/update/snapshot.py 中 _DeleteFiles._compute_deletes 的大约第 598 行开始,跟踪 delete_data_file() 如何记录显式指定的文件。运行 issue 中的复现,然后验证显式引用的数据文件会在 commit 后被删除,同时基于 predicate 的删除仍然有效。当表为空而不是保留 [1, 2, 3] 时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
databases
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
活跃
描述清晰度
描述清楚
新手友好度
72/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。