apache / apache/iceberg

Core: Add metadata table for discovering metadata tables

Open
#17,132 3 comments 1 reaction 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Feature Request / Improvement

### Feature request

Iceberg exposes many useful metadata tables, such as `files`, `data_files`, `delete_files`, `manifests`, `snapshots`, `history`, `metadata_log_entries`, `refs`, and others.

I would like to propose adding a new metadata table that lists the metadata tables available for a given Iceberg table.

### Motivation / use case
Metadata tables are useful for debugging, observability, table maintenance, and understanding Iceberg table internals. For example, users can inspect files, manifests, snapshots, refs, and metadata log entries directly through SQL.

However, users currently need to know these metadata table names ahead of time from documentation, examples, or source code. It would be helpful to make metadata tables self-discoverable from SQL.

A self-describing metadata table would make Iceberg metadata easier to explore without adding engine-specific SQL syntax such as:

```sql
SHOW METADATA TABLES FOR catalog.db.table;
```

Instead, this would reuse Iceberg's existing metadata table model.
For example, in Spark:

```sql
SELECT * FROM catalog.db.table.metadata_tables;
```
Excepted output:
```
entries
files
data_files
delete_files
history
metadata_log_entries
snapshots
refs
manifests
partitions
all_data_files
all_delete_files
all_files
all_manifests
all_entries
position_deletes
metadata_tables
```

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Begin with Iceberg’s existing metadata-table model and Spark integration, then compare how the listed tables are exposed. Define the schema and behavior for a table that reports available metadata tables; done means the documented SQL query returns the expected names consistently, including the new table if appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.