Partial-progress compaction can lose manifest-list encryption keys
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Concurrent partial-progress commits on an encrypted Hive table can commit a snapshot without the wrapped key for its manifest list. The rewrite reports failed groups and returns, but the snapshot's key is lost, leaving the table unreadable to users and seemingly lost. They see:
```
java.util.concurrent.ExecutionException: java.lang.NullPointerException:
Cannot invoke "org.apache.iceberg.encryption.EncryptedKey.encryptedKeyMetadata()"
because "manifestListKey" is null
Caused by: java.lang.NullPointerException
at org.apache.iceberg.encryption.EncryptionUtil.decryptManifestListKeyMetadata(...)
at org.apache.iceberg.BaseSnapshot.allManifests(...)
at org.apache.iceberg.FastAppend.apply(...)
at org.apache.iceberg.SnapshotProducer.commit(...)
```
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by tracing the partial-progress commit path into FastAppend.apply and SnapshotProducer.commit, then inspect BaseSnapshot.allManifests and EncryptionUtil.decryptManifestListKeyMetadata around manifestListKey handling. Reproduce the concurrent encrypted Hive-table rewrite scenario and add coverage showing that a partial-progress commit does not produce a snapshot missing its manifest-list key.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- databases, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100