apache / apache/iceberg

Encryption keys in metadata are not cleaned up

Open
#16,352 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.1 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

We have an encrypted Iceberg Table. It has high number of writes which leads to many snapshots, we run expire_snapshots aggressively, but overtime we notice the metadata file keeps on growing. Which leads to reduced table performance.

We have metadata that is 90%+ unused encryption keys. 40MB+ of encryption keys 😢 .

The bug seems to be that `encryption-keys` within the metadata file are not cleaned up.

In both expire_snapshots which calls into [removeSnapshotsInternal](https://github.com/apache/iceberg/blob/3166d6c3604b3599abeb6012375c2440dd5aaa35/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1426).

and also with expireSnapshots with `cleanExpiredMetadata` set to `true`. Which calls into: [cleanExpiredMetadata](https://github.com/apache/iceberg/blob/5cc19891a25f314971bcbc3ee9d6d4fa52d942de/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java#L239-L238).

Neither of these cleanup encryption keys and should be updated to clean up encryption keys.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.