Core: Add metadata tables for Puffin file references
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
## Motivation
Iceberg stores several types of metadata in Puffin files, including table statistics and deletion vectors. These files are referenced through different parts of Iceberg metadata:
* Statistics files are registered in table metadata.
* Deletion-vector Puffin files are referenced by live delete-file entries in delete manifests.
There is currently no metadata table that provides a unified view of the Puffin file references associated with a table snapshot. Users must inspect table metadata JSON, manifest lists, and delete manifests separately to answer questions such as:
* Which Puffin files are referenced by the current or a historical snapshot?
* Which references come from registered statistics and which come from deletion vectors?
* Which snapshots reference the same physical Puffin file?
* How many distinct Puffin blobs in a file are referenced by a snapshot?
* Which blob types and table fields are associated with those references?
## Proposal
Add two metadata tables:
### `puffin_files`
Exposes Puffin file references associated with one selected snapshot.
The selected snapshot is the table's current snapshot by default and may be changed using the standard metadata-table time-travel APIs.
### `all_puffin_files`
Exposes Puffin file references associated with all snapshots currently retained in table metadata.
This follows the existing relationship between snapshot-scoped and all-snapshot metadata tables.
## Metadata sources
The initial implementation supports two metadata sources:
* `statistics`: statistics files currently registered in table metadata
* `deletion_vector`: Puffin files referenced by live deletion-vector entries in delete manifests
Additional Puffin-backed metadata sources may be added in the future without changing the basic table model.
## Row semantics
Each row summarizes the Puffin blob references for one tuple of:
* reference snapshot ID
* metadata source
* physical Puffin file path
A physical Puffin file may therefore appear in multiple rows when:
* it is referenced by multiple retained snapshots, or
* it is referenced through multiple metadata sources.
The blob count reports the number of distinct blobs referenced by that row. It is not necessarily the total number of blobs physically stored in the Puffin file.
## Proposed columns
The two tables expose equivalent information, using `snapshot_id` for the snapshot-scoped table and `reference_snapshot_id` for the all-snapshots table:
* `snapshot_id` / `reference_snapshot_id`
* `file_path`
* `source`
* `file_size_in_bytes`
* `referenced_blob_count`
* `referenced_blob_types`
* `referenced_fields`
Each element of `referenced_fields` contains:
* `field_id`
* `current_field_name`
Field IDs are authoritative. Field names are resolved against the current table schema and may be null when a referenced field has been dropped.
## Snapshot and history semantics
Deletion-vector references are reconstructed independently for each retained snapshot from that snapshot's live delete manifests.
Statistics files are obtained from the statistics files currently registered in table metadata and are associated with their registered snapshot IDs.
Statistics registration may be changed independently of table snapshots. The metadata tables do not reconstruct historical statistics-registration state from previous metadata files. A statistics file that was replaced or removed is not reported merely because its associated data snapshot is still retained.
If multiple branches or tags reference the same snapshot, `all_puffin_files` reports the Puffin references once for that snapshot ID rather than once per named reference.
## Planning
The `all_puffin_files` table should support conservative task pruning for predicates over:
* `reference_snapshot_id`
* `source`
A filter such as `source = 'statistics'` should avoid reading delete manifests. Snapshot predicates should avoid creating tasks for snapshots that cannot match.
Unknown predicates must not cause incorrect pruning, including when they appear under `NOT`.
## Non-goals
This proposal does not:
* open Puffin files;
* read Puffin footers;
* parse or validate Puffin blob payloads;
* expose blobs that are not referenced by Iceberg metadata;
* reconstruct historical statistics-registration state;
* verify whether referenced physical files currently exist;
* identify files that are safe to delete;
* modify statistics or deletion-vector metadata.
All reported information is derived from Iceberg table metadata, snapshot manifest lists, and delete manifests.
## Implementation
* [ ] #17192 — Add the snapshot-scoped `puffin_files` metadata table
* [ ] #17282 — Add the all-snapshots `all_puffin_files` metadata table
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start by reading implementation issues #17192 and #17282, then review the existing metadata-table time-travel APIs and the table metadata, manifest-list, and delete-manifest sources named here. Done means both tables expose the specified reference rows and columns, preserve the stated snapshot semantics, and conservatively prune all-snapshot tasks by reference_snapshot_id and source.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100