apache / apache/iceberg

Partial-progress compaction can lose manifest-list encryption keys

Open
#17,989 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

Concurrent partial-progress commits on an encrypted Hive table can commit a snapshot without the wrapped key for its manifest list. The rewrite reports failed groups and returns, but the snapshot's key is lost, leaving the table unreadable to users and seemingly lost. They see:

```
java.util.concurrent.ExecutionException: java.lang.NullPointerException:
Cannot invoke "org.apache.iceberg.encryption.EncryptedKey.encryptedKeyMetadata()"
because "manifestListKey" is null

Caused by: java.lang.NullPointerException
at org.apache.iceberg.encryption.EncryptionUtil.decryptManifestListKeyMetadata(...)
at org.apache.iceberg.BaseSnapshot.allManifests(...)
at org.apache.iceberg.FastAppend.apply(...)
at org.apache.iceberg.SnapshotProducer.commit(...)
```

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by tracing the partial-progress commit path into FastAppend.apply and SnapshotProducer.commit, then inspect BaseSnapshot.allManifests and EncryptionUtil.decryptManifestListKeyMetadata around manifestListKey handling. Reproduce the concurrent encrypted Hive-table rewrite scenario and add coverage showing that a partial-progress commit does not produce a snapshot missing its manifest-list key.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.