apache / apache/iceberg-python
_DeleteFiles.delete_data_file() silently drops explicit file references
- Lingua principale
- Python
- Stelle
- 1.1k
- Fork
- 581
- Merge medio
- 1g 17h
- PR unite (30g)
- 78
Descrizione
## Description
`_DeleteFiles.delete_data_file()` silently drops explicit file references because `_compute_deletes` resets `self._deleted_data_files = set()` before scanning manifests by predicate.
The `delete_data_file()` method is inherited from `_SnapshotProducer` and adds to `self._deleted_data_files`. However, when `_DeleteFiles._compute_deletes` runs (triggered by `_deleted_entries()`), it resets this field to an empty set and repopulates it only from files matched by the predicate evaluator. Any explicitly added files are silently lost.
While the high-level `table.delete()` API only uses predicates (so this isn't hit in normal usage), the low-level `update_snapshot().delete()` API exposes `delete_data_file()` as a public method. Calling it produces no error and no effect.
## Reproduction
```python
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField
catalog = load_catalog("default")
catalog.create_namespace("default")
table = catalog.create_table(
"default.delete_explicit",
Schema(NestedField(1, "x", LongType(), required=False)),
)
table.append(pa.table({"x": [1, 2, 3]}))
data_file = next(iter(table.scan().plan_files())).file
# This should delete the file but silently does nothing
with table.transaction() as tx:
delete_snapshot = tx.update_snapshot().delete()
delete_snapshot.delete_data_file(data_file)
# Bug: table still has [1, 2, 3]
print(table.scan().to_arrow()["x"].to_pylist())
```
Expected: table is empty after commit.
Actual: table still has `[1, 2, 3]`.
## Root cause
In `pyiceberg/table/update/snapshot.py`, `_DeleteFiles._compute_deletes` (line ~598):
```python
self._deleted_data_files = set() # <-- overwrites any explicit files added via delete_data_file()
```
Then it repopulates from predicate matches only. Since no predicate was set (`AlwaysFalse` by default), nothing matches, nothing is deleted.
## Suggested fix
Before resetting, preserve explicit files and include them in the deletion scan:
```python
# Preserve files explicitly requested for deletion
explicit_deletes = set(self._deleted_data_files)
self._deleted_data_files = set()
# ... existing predicate-based scan ...
# After the scan, also mark explicitly requested files as deleted:
for entry in manifest_entries:
if entry.data_file in explicit_deletes:
# mark as deleted
```
Alternatively, prevent calling `delete_data_file()` on `_DeleteFiles` entirely by raising `NotImplementedError`.
## Related
Follow-up observation from #3818 review. The `_OverwriteFiles` path handles `delete_data_file()` correctly because it does not reset the field.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Inizia in pyiceberg/table/update/snapshot.py, in _DeleteFiles._compute_deletes, intorno alla riga 598, e segui il modo in cui delete_data_file() registra i file specificati esplicitamente. Esegui la riproduzione dell’issue, quindi verifica che un file di dati referenziato esplicitamente venga eliminato dopo il commit, mentre l’eliminazione basata su predicato continui a funzionare. Il lavoro è completato quando la tabella è vuota invece di conservare [1, 2, 3].
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- databases
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Attiva
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 72/100