apache / apache/iceberg-python

_DeleteFiles.delete_data_file() silently drops explicit file references

Aperta
#3,857 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
1.1k
Fork
581
Merge medio
1g 17h
PR unite (30g)
78

Descrizione

## Description

`_DeleteFiles.delete_data_file()` silently drops explicit file references because `_compute_deletes` resets `self._deleted_data_files = set()` before scanning manifests by predicate.

The `delete_data_file()` method is inherited from `_SnapshotProducer` and adds to `self._deleted_data_files`. However, when `_DeleteFiles._compute_deletes` runs (triggered by `_deleted_entries()`), it resets this field to an empty set and repopulates it only from files matched by the predicate evaluator. Any explicitly added files are silently lost.

While the high-level `table.delete()` API only uses predicates (so this isn't hit in normal usage), the low-level `update_snapshot().delete()` API exposes `delete_data_file()` as a public method. Calling it produces no error and no effect.

## Reproduction

```python
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField

catalog = load_catalog("default")
catalog.create_namespace("default")
table = catalog.create_table(
"default.delete_explicit",
Schema(NestedField(1, "x", LongType(), required=False)),
)
table.append(pa.table({"x": [1, 2, 3]}))

data_file = next(iter(table.scan().plan_files())).file

# This should delete the file but silently does nothing
with table.transaction() as tx:
delete_snapshot = tx.update_snapshot().delete()
delete_snapshot.delete_data_file(data_file)

# Bug: table still has [1, 2, 3]
print(table.scan().to_arrow()["x"].to_pylist())
```

Expected: table is empty after commit.
Actual: table still has `[1, 2, 3]`.

## Root cause

In `pyiceberg/table/update/snapshot.py`, `_DeleteFiles._compute_deletes` (line ~598):

```python
self._deleted_data_files = set() # <-- overwrites any explicit files added via delete_data_file()
```

Then it repopulates from predicate matches only. Since no predicate was set (`AlwaysFalse` by default), nothing matches, nothing is deleted.

## Suggested fix

Before resetting, preserve explicit files and include them in the deletion scan:

```python
# Preserve files explicitly requested for deletion
explicit_deletes = set(self._deleted_data_files)
self._deleted_data_files = set()

# ... existing predicate-based scan ...
# After the scan, also mark explicitly requested files as deleted:
for entry in manifest_entries:
if entry.data_file in explicit_deletes:
# mark as deleted
```

Alternatively, prevent calling `delete_data_file()` on `_DeleteFiles` entirely by raising `NotImplementedError`.

## Related

Follow-up observation from #3818 review. The `_OverwriteFiles` path handles `delete_data_file()` correctly because it does not reset the field.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia in pyiceberg/table/update/snapshot.py, in _DeleteFiles._compute_deletes, intorno alla riga 598, e segui il modo in cui delete_data_file() registra i file specificati esplicitamente. Esegui la riproduzione dell’issue, quindi verifica che un file di dati referenziato esplicitamente venga eliminato dopo il commit, mentre l’eliminazione basata su predicato continui a funzionare. Il lavoro è completato quando la tabella è vuota invece di conservare [1, 2, 3].

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
databases
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Attiva
Chiarezza
Specificata chiaramente
Idoneità per principianti
72/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.