apache / apache/arrow-rs

[Parquet] Support async Parquet encryption key retrieval

Open
#8,238 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

The current [KeyRetriever](https://github.com/apache/arrow-rs/blob/1dacecba8e11cac307eea5d1a0f10c22d7f4a8b7/parquet/src/encryption/decrypt.rs#L102) trait is synchronous, but it is common for implementations to need to make network calls when decrypting keys, for example to a key management service (KMS). Currently this will require blocking a thread which is not desirable in an async application.

For compatibility with the synchronous API, we should continue to allow the use of a synchronous `KeyRetriever`.

**Describe the solution you'd like**

My proposed solution is to add a new interface, `AsyncKeyRetriever`, and allow `FileDecryptionProperties` to hold either a `KeyRetriever` or an `AsyncKeyRetriever`. New code paths will be added to use the async retriever. When used via synchronous code, an error will be raised if an `AsyncKeyRetriever` has been set.

I'm concerned this might require duplicating a lot of existing synchronous code, but hopefully that can be minimized.

**Describe alternatives you've considered**

* Don't change anything and require blocking. The impact of this may not be too bad when using double wrapping, so interactions with the KMS are infrequent.
* Make `KeyRetriever` async and don't allow using it from sync code (suggested in https://github.com/apache/arrow-rs/pull/7387#issuecomment-2789490252 by @tustvold). This is a bit tidier by not having two similar traits, but I don't think it would simplify the implementation all that much and it would be nice to continue to allow a synchronous `KeyRetriever` for users using the synchronous APIs.

Contributor guide

Open the contributing guide

Research direction

Start with parquet/src/encryption/decrypt.rs, especially the KeyRetriever trait, and trace how FileDecryptionProperties stores and uses it. Review the synchronous decryption paths and determine where an AsyncKeyRetriever must be supported while preserving synchronous KeyRetriever behavior. Done means async decryption can retrieve keys without blocking, while synchronous use reports an error when given an async retriever.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.