apache / apache/arrow-go

[Go][Parquet] reader panics when selectively decrypting columns

Open
#69 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Type: bug
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Hello, following up on issue apache/arrow#14940, I am leveraging the go arrow library to encrypt/decrypt parquet files.

My goal is to read parquet files with encrypted columns, decrypt some of the columns and produce a new parquet file with the decrypted column while keeping the encrypted column as part of the file.

Unfortunately, when reading parquet files with an encryption key per column, the reader panics with the error hidden column exception if one column is missing from the decryption configuration.

To reproduce this behavior, I have forked the repo and added a decryption configuration (no 5), which you can try here: https://github.com/ahmb84/arrow/blob/bf1964552ca4d62988ca306f4b98db08e18ac7d8/go/parquet/encryption_read_config_test.go#L158

I've then tried to implement a version of the reader that can bypass the encryption context if no key is present in the decryption configuration for a column encrypted with its key.
It seems to be working, as the reader does not panic, but the encrypted column appears to be skipped as no values are read (I believe that's the expected behavior?).

Ideally, the reader would be able to return the encrypted value so that I could rebuild a parquet file with the encrypted data and some decrypted data.
You can find my implementation here: https://github.com/ahmb84/arrow/tree/selective-column-decryption-impl

Let me know if my understanding is correct. I appreciate your help!

### Component(s)

Go, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.