Concurrent commits through one `Table` can lose manifest-list encryption keys
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
None
### Please describe the bug 🐞
Concurrent commits through one shared, encrypted `Table` can commit a snapshot without the wrapped key required to read its manifest list. The commit succeeds, but the snapshot's key is lost, leaving the table unreadable to users and seemingly lost.
See https://github.com/apache/iceberg/pull/17982 for a reproduction, the user-facing error will be something like:
```
java.util.concurrent.ExecutionException: java.lang.NullPointerException:
Cannot invoke "org.apache.iceberg.encryption.EncryptedKey.encryptedKeyMetadata()"
because "manifestListKey" is null
Caused by: java.lang.NullPointerException
at org.apache.iceberg.encryption.EncryptionUtil.decryptManifestListKeyMetadata(...)
at org.apache.iceberg.BaseSnapshot.allManifests(...)
at org.apache.iceberg.FastAppend.apply(...)
at org.apache.iceberg.SnapshotProducer.commit(...)
```
This situation surfaces in https://github.com/apache/iceberg/issues/17989, but also more explicitly in https://github.com/apache/iceberg/blob/main/hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveTableConcurrency.java.
The root cause is that https://github.com/apache/iceberg/blob/main/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java has race conditions which, for encryption, can lead to key loss. The issue here is that a writer can write a snapshot, mutating its encryption manager, when a concurrent refresh from another writer "installs" a different manager before the first writer's commit, losing the first writer's snapshot keys. Concretely, with two writers:
1. Writer A refreshes the table, constructs its manifest-list writer with encryption manager A, and blocks while that manager unwraps its key-encryption key before publishing the new manifest-list key.
2. Writer B refreshes the same table operations, "installs" encryption manager B, constructs its writer, and blocks at the same point.
3. Writer A resumes and publishes the new manifest-list key to manager A.
4. Writer A's Hive commit copies encryption keys from *manager B*.
5. Writer A's snapshot commits with a manifest-list key ID, but its key is absent from table metadata.
6. Writer B resumes. Its first commit is stale, and its retry fails in `FastAppend.apply` while reading writer A's manifest list because `manifestListKey` is null.
7. After reloading the table from the catalog, writer A's snapshot still lacks its key and planning it therefore fails with the same exception.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with HiveTableOperations.java and the concurrency scenarios in hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveTableConcurrency.java. Follow how concurrent refresh and commit operations install encryption managers and copy keys, then use the reproduction in PR 17982 to verify the committed snapshot retains its manifest-list key and remains readable after reload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100