apache / apache/parquet-java

Sample of usage Parquet-1396 and Parquet-1178 for column level encryption with pluggable key access

Open
#2,221 1 comment 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: test
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

This Jira provides a sample to use Parquet-1396 and Parquet-1178 column level encryption with pluggable key access. The Spark SQL application shows how to configure Parquet-1396 to encrypt and columns. The project CryptoMetadataRetriever shows how to implement the interface defined in Parquet-1396 as an example. 

The project to be uploaded soon. 

 

**Reporter**: [Xinli Shang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shangx@uber.com) / @shangxinli
#### Related issues:
- [EncryptionPropertiesFactory and DecryptionPropertiesFactory](https://github.com/apache/parquet-java/issues/2220) (Blocked)
- [Passing Field Metadata to Parquet](https://issues.apache.org/jira/browse/SPARK-25858) (Blocked)
- [Parquet modular encryption](https://github.com/apache/parquet-java/issues/2110) (depends upon)
#### Externally tracked issue: [https://github.com/apache/parquet-java/issues/2220](https://github.com/apache/parquet-java/issues/2220)

**Note**: *This issue was originally created as [PARQUET-1397](https://issues.apache.org/jira/browse/PARQUET-1397). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading Parquet-1396 and Parquet-1178, then review the blocked related issues 2220 and SPARK-25858 and the dependent issue 2110. The issue says a Spark SQL application and a CryptoMetadataRetriever project should demonstrate encryption configuration and the interface implementation, but the project is not yet attached. Done means the sample project is uploaded and documents both usage paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, security
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.