Core: Add metadata table for discovering metadata tables
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Feature Request / Improvement
### Feature request
Iceberg exposes many useful metadata tables, such as `files`, `data_files`, `delete_files`, `manifests`, `snapshots`, `history`, `metadata_log_entries`, `refs`, and others.
I would like to propose adding a new metadata table that lists the metadata tables available for a given Iceberg table.
### Motivation / use case
Metadata tables are useful for debugging, observability, table maintenance, and understanding Iceberg table internals. For example, users can inspect files, manifests, snapshots, refs, and metadata log entries directly through SQL.
However, users currently need to know these metadata table names ahead of time from documentation, examples, or source code. It would be helpful to make metadata tables self-discoverable from SQL.
A self-describing metadata table would make Iceberg metadata easier to explore without adding engine-specific SQL syntax such as:
```sql
SHOW METADATA TABLES FOR catalog.db.table;
```
Instead, this would reuse Iceberg's existing metadata table model.
For example, in Spark:
```sql
SELECT * FROM catalog.db.table.metadata_tables;
```
Excepted output:
```
entries
files
data_files
delete_files
history
metadata_log_entries
snapshots
refs
manifests
partitions
all_data_files
all_delete_files
all_files
all_manifests
all_entries
position_deletes
metadata_tables
```
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Begin with Iceberg’s existing metadata-table model and Spark integration, then compare how the listed tables are exposed. Define the schema and behavior for a table that reports available metadata tables; done means the documented SQL query returns the expected names consistently, including the new table if appropriate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100