apache / apache/iceberg-python

_DeleteFiles.delete_data_file() silently drops explicit file references

オープン
#3,857 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
1.1k
フォーク
581
平均マージ
1日 17時間
マージ済み PR(30日)
78

説明

## Description

`_DeleteFiles.delete_data_file()` silently drops explicit file references because `_compute_deletes` resets `self._deleted_data_files = set()` before scanning manifests by predicate.

The `delete_data_file()` method is inherited from `_SnapshotProducer` and adds to `self._deleted_data_files`. However, when `_DeleteFiles._compute_deletes` runs (triggered by `_deleted_entries()`), it resets this field to an empty set and repopulates it only from files matched by the predicate evaluator. Any explicitly added files are silently lost.

While the high-level `table.delete()` API only uses predicates (so this isn't hit in normal usage), the low-level `update_snapshot().delete()` API exposes `delete_data_file()` as a public method. Calling it produces no error and no effect.

## Reproduction

```python
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField

catalog = load_catalog("default")
catalog.create_namespace("default")
table = catalog.create_table(
"default.delete_explicit",
Schema(NestedField(1, "x", LongType(), required=False)),
)
table.append(pa.table({"x": [1, 2, 3]}))

data_file = next(iter(table.scan().plan_files())).file

# This should delete the file but silently does nothing
with table.transaction() as tx:
delete_snapshot = tx.update_snapshot().delete()
delete_snapshot.delete_data_file(data_file)

# Bug: table still has [1, 2, 3]
print(table.scan().to_arrow()["x"].to_pylist())
```

Expected: table is empty after commit.
Actual: table still has `[1, 2, 3]`.

## Root cause

In `pyiceberg/table/update/snapshot.py`, `_DeleteFiles._compute_deletes` (line ~598):

```python
self._deleted_data_files = set() # <-- overwrites any explicit files added via delete_data_file()
```

Then it repopulates from predicate matches only. Since no predicate was set (`AlwaysFalse` by default), nothing matches, nothing is deleted.

## Suggested fix

Before resetting, preserve explicit files and include them in the deletion scan:

```python
# Preserve files explicitly requested for deletion
explicit_deletes = set(self._deleted_data_files)
self._deleted_data_files = set()

# ... existing predicate-based scan ...
# After the scan, also mark explicitly requested files as deleted:
for entry in manifest_entries:
if entry.data_file in explicit_deletes:
# mark as deleted
```

Alternatively, prevent calling `delete_data_file()` on `_DeleteFiles` entirely by raising `NotImplementedError`.

## Related

Follow-up observation from #3818 review. The `_OverwriteFiles` path handles `delete_data_file()` correctly because it does not reset the field.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

pyiceberg/table/update/snapshot.py の _DeleteFiles._compute_deletes(598 行目付近)から始め、delete_data_file() が明示的に指定されたファイルをどのように記録するかを追跡します。issue の再現手順を実行し、その後、明示的に参照されたデータファイルが commit 後に削除されることを確認します。同時に、predicate ベースの削除が引き続き機能することも確認します。テーブルが [1, 2, 3] を保持したままではなく空になれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
databases
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
活発
明瞭さ
明確に書かれている
初心者へのやさしさ
72/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。