Fix reading Encrypted Deletion Vectors
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
None
### Query engine
None
### Please describe the bug 🐞
When using **Table Level Encryption**, If there are `Deletion Vectors` the Read fails
```
Caused by:
java.lang.IllegalStateException: Null key metadata buffer
at org.apache.iceberg.encryption.StandardKeyMetadata.castOrParse(StandardKeyMetadata.java:111)
at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.keyMetadata(StandardEncryptionManager.java:302)
at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.decrypted(StandardEncryptionManager.java:313)
at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.location(StandardEncryptionManager.java:333)
at org.apache.iceberg.encryption.EncryptingFileIO.bulkDecrypt(EncryptingFileIO.java:71)
at org.apache.iceberg.spark.source.BaseReader.inputFiles(BaseReader.java:186)
at org.apache.iceberg.spark.source.BaseReader.getInputFile(BaseReader.java:180)
at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:108)
at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:43)
at org.apache.iceberg.spark.source.BaseReader.next(BaseReader.java:144)
at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:148)
at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:186)
```
OR:
```
Caused by: java.lang.IllegalArgumentException: Invalid bitmap data length: 4096, expected 36
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:321)
at org.apache.iceberg.deletes.BitmapPositionDeleteIndex.readBitmapDataLength(BitmapPositionDeleteIndex.java:184)
at org.apache.iceberg.deletes.BitmapPositionDeleteIndex.deserialize(BitmapPositionDeleteIndex.java:148)
at org.apache.iceberg.deletes.PositionDeleteIndex.deserialize(PositionDeleteIndex.java:113)
at org.apache.iceberg.data.BaseDeleteLoader.readDV(BaseDeleteLoader.java:179)
at org.apache.iceberg.data.BaseDeleteLoader.loadPositionDeletes(BaseDeleteLoader.java:165)
at org.apache.iceberg.data.DeleteFilter.deletedRowPositions(DeleteFilter.java:255)
at org.apache.iceberg.data.DeleteFilter.applyPosDeletes(DeleteFilter.java:266)
at org.apache.iceberg.data.DeleteFilter.filter(DeleteFilter.java:189)
```
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by tracing encrypted deletion-vector reads through EncryptingFileIO.bulkDecrypt, StandardEncryptionManager.StandardDecryptedInputFile, and BaseDeleteLoader.readDV, using the reported Table Level Encryption scenario. Compare the key-metadata and bitmap deserialization paths; done means reading a table with deletion vectors succeeds without either reported exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- databases, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100