Encryption keys in metadata are not cleaned up
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
We have an encrypted Iceberg Table. It has high number of writes which leads to many snapshots, we run expire_snapshots aggressively, but overtime we notice the metadata file keeps on growing. Which leads to reduced table performance.
We have metadata that is 90%+ unused encryption keys. 40MB+ of encryption keys 😢 .
The bug seems to be that `encryption-keys` within the metadata file are not cleaned up.
In both expire_snapshots which calls into [removeSnapshotsInternal](https://github.com/apache/iceberg/blob/3166d6c3604b3599abeb6012375c2440dd5aaa35/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1426).
and also with expireSnapshots with `cleanExpiredMetadata` set to `true`. Which calls into: [cleanExpiredMetadata](https://github.com/apache/iceberg/blob/5cc19891a25f314971bcbc3ee9d6d4fa52d942de/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java#L239-L238).
Neither of these cleanup encryption keys and should be updated to clean up encryption keys.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Assessment
This issue has not been assessed yet.