apache / apache/iceberg-python

_DeleteFiles.delete_data_file() silently drops explicit file references

Offen
#3,857 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
1.1k
Forks
581
Ø Merge
1 T. 17 Std.
Gemergte PRs (30 T.)
77

Beschreibung

## Description

`_DeleteFiles.delete_data_file()` silently drops explicit file references because `_compute_deletes` resets `self._deleted_data_files = set()` before scanning manifests by predicate.

The `delete_data_file()` method is inherited from `_SnapshotProducer` and adds to `self._deleted_data_files`. However, when `_DeleteFiles._compute_deletes` runs (triggered by `_deleted_entries()`), it resets this field to an empty set and repopulates it only from files matched by the predicate evaluator. Any explicitly added files are silently lost.

While the high-level `table.delete()` API only uses predicates (so this isn't hit in normal usage), the low-level `update_snapshot().delete()` API exposes `delete_data_file()` as a public method. Calling it produces no error and no effect.

## Reproduction

```python
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField

catalog = load_catalog("default")
catalog.create_namespace("default")
table = catalog.create_table(
"default.delete_explicit",
Schema(NestedField(1, "x", LongType(), required=False)),
)
table.append(pa.table({"x": [1, 2, 3]}))

data_file = next(iter(table.scan().plan_files())).file

# This should delete the file but silently does nothing
with table.transaction() as tx:
delete_snapshot = tx.update_snapshot().delete()
delete_snapshot.delete_data_file(data_file)

# Bug: table still has [1, 2, 3]
print(table.scan().to_arrow()["x"].to_pylist())
```

Expected: table is empty after commit.
Actual: table still has `[1, 2, 3]`.

## Root cause

In `pyiceberg/table/update/snapshot.py`, `_DeleteFiles._compute_deletes` (line ~598):

```python
self._deleted_data_files = set() # <-- overwrites any explicit files added via delete_data_file()
```

Then it repopulates from predicate matches only. Since no predicate was set (`AlwaysFalse` by default), nothing matches, nothing is deleted.

## Suggested fix

Before resetting, preserve explicit files and include them in the deletion scan:

```python
# Preserve files explicitly requested for deletion
explicit_deletes = set(self._deleted_data_files)
self._deleted_data_files = set()

# ... existing predicate-based scan ...
# After the scan, also mark explicitly requested files as deleted:
for entry in manifest_entries:
if entry.data_file in explicit_deletes:
# mark as deleted
```

Alternatively, prevent calling `delete_data_file()` on `_DeleteFiles` entirely by raising `NotImplementedError`.

## Related

Follow-up observation from #3818 review. The `_OverwriteFiles` path handles `delete_data_file()` correctly because it does not reset the field.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Beginne in pyiceberg/table/update/snapshot.py bei _DeleteFiles._compute_deletes, ungefähr bei Zeile 598, und verfolge, wie delete_data_file() explizit angegebene Dateien erfasst. Führe die Reproduktion aus dem Issue aus und überprüfe anschließend, dass eine explizit referenzierte Datendatei nach dem Commit gelöscht wird, während die prädikatbasierte Löschung weiterhin funktioniert. Als erledigt gilt die Aufgabe, wenn die Tabelle leer ist, statt [1, 2, 3] zu behalten.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
databases
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Aktiv
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
72/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.