apache / apache/iceberg-python

Remove deleted data files with expire_snapshots

Abierto
#2,604 5 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
1.1k
Forks
581
Merge medio
1 d 17 h
PR fusionados (30 d)
78

Descripción

### Feature Request / Improvement

Running an expire snapshots operation will only rewrite the metadata file without the expired snapshots (and refs/ statistics). It does not delete deleted data files referenced only by the expired snapshots. This can be observed by deleting an entire table and calling `expire_snapshots` - the data files still exist. Trino and spark both clean up deleted data files when all snapshots referencing them are expired.

From the spec:
```
When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID
in which the file was deleted and status 2 (deleted). The file may be deleted from the file system
when the snapshot in which it was deleted is garbage collected, assuming that older snapshots
have also been garbage collected [1].
...
1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is
garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It
is easier to track what files are deleted in a snapshot and delete them when that snapshot expires.
It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge
cases where incremental deletes can break table snapshots.
```

Happy to work on this if others agree that it should be added :)

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Comienza rastreando la operación expire_snapshots y cómo reescribe los metadatos; después, inspecciona el manejo de las entradas de manifest marcadas como eliminadas. Confirma el comportamiento frente al caso descrito de eliminación de toda la tabla y las pruebas existentes relacionadas; el trabajo estará terminado cuando se eliminen los archivos de datos a los que solo hacen referencia los snapshots expirados, mientras que permanezcan los archivos que aún necesitan los snapshots conservados.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
databases
Tipo de issue
Nueva funcionalidad
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.