apache / apache/iceberg

Fix reading Encrypted Deletion Vectors

Open
#16,157 1 comment 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

None

### Query engine

None

### Please describe the bug 🐞

When using **Table Level Encryption**, If there are `Deletion Vectors` the Read fails
```
Caused by:
java.lang.IllegalStateException: Null key metadata buffer
at org.apache.iceberg.encryption.StandardKeyMetadata.castOrParse(StandardKeyMetadata.java:111)
at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.keyMetadata(StandardEncryptionManager.java:302)
at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.decrypted(StandardEncryptionManager.java:313)
at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.location(StandardEncryptionManager.java:333)
at org.apache.iceberg.encryption.EncryptingFileIO.bulkDecrypt(EncryptingFileIO.java:71)
at org.apache.iceberg.spark.source.BaseReader.inputFiles(BaseReader.java:186)
at org.apache.iceberg.spark.source.BaseReader.getInputFile(BaseReader.java:180)
at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:108)
at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:43)
at org.apache.iceberg.spark.source.BaseReader.next(BaseReader.java:144)
at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:148)
at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:186)
```
OR:
```
Caused by: java.lang.IllegalArgumentException: Invalid bitmap data length: 4096, expected 36
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:321)
at org.apache.iceberg.deletes.BitmapPositionDeleteIndex.readBitmapDataLength(BitmapPositionDeleteIndex.java:184)
at org.apache.iceberg.deletes.BitmapPositionDeleteIndex.deserialize(BitmapPositionDeleteIndex.java:148)
at org.apache.iceberg.deletes.PositionDeleteIndex.deserialize(PositionDeleteIndex.java:113)
at org.apache.iceberg.data.BaseDeleteLoader.readDV(BaseDeleteLoader.java:179)
at org.apache.iceberg.data.BaseDeleteLoader.loadPositionDeletes(BaseDeleteLoader.java:165)
at org.apache.iceberg.data.DeleteFilter.deletedRowPositions(DeleteFilter.java:255)
at org.apache.iceberg.data.DeleteFilter.applyPosDeletes(DeleteFilter.java:266)
at org.apache.iceberg.data.DeleteFilter.filter(DeleteFilter.java:189)
```

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by tracing encrypted deletion-vector reads through EncryptingFileIO.bulkDecrypt, StandardEncryptionManager.StandardDecryptedInputFile, and BaseDeleteLoader.readDV, using the reported Table Level Encryption scenario. Compare the key-metadata and bitmap deserialization paths; done means reading a table with deletion vectors succeeds without either reported exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.