apache / apache/iceberg

Allow `write.metadata.previous-versions-max` table property to be 0 for empty metadata logs

Open
#16,797 4 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

`write.metadata.previous-versions-max` is currently floored to 1:

https://github.com/apache/iceberg/blob/70b90330582e4023a097330620a3e63746b2375b/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1801-L1807

so a table always retains at least one previous metadata file in the `metadata-log`.

I was a bit confused by this, because I don't see a concrete reason to disallow _empty_ metadata logs; I'd therefore like to know what folks think about permitting a value of 0 instead to enable this.

With [`write.metadata.delete-after-commit.enabled=true`](https://iceberg.apache.org/docs/latest/maintenance/#remove-old-metadata-files), this would then delete the superseded metadata file on every commit.

(My use case for this is from a REST catalog principle-based standpoint where an empty metadata log would allow metadata JSON to be "self-contained" and only reference its current snapshots, instead of indirectly reference snapshots in prior metadata files too)

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start in core/src/main/java/org/apache/iceberg/TableMetadata.java at the linked lines that floor write.metadata.previous-versions-max to 1. Trace the metadata-log retention behavior and its tests, including the interaction with write.metadata.delete-after-commit.enabled. Done means a value of 0 permits an empty metadata log and the described deletion behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.