apache / apache/arrow

[Python][Parquet][PyArrow] Cannot save footer files outside the parquet file

Open
#35,678 2 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

Hi Team,

I have a use case where I'm trying to encrypt my parquet files and save the keys externally so it's easier to rotate the master keys and re-encrypt the DEK without reading the original file.

However, I'm unable to save the footer files separately with just specifying the `internal_key_material=False` in `EncryptionConfiguration`. Is there something else missing in the documentation or is this is a bug ?

Here's what I'm doing:

```
encryption_config = pe.EncryptionConfiguration(
footer_key=key_name,
column_keys={
key_name: ["a", "b"],
},
plaintext_footer=True,
internal_key_material=False,
data_key_length_bits=256,
)
```

```
file_encryption_properties = crypto_factory.file_encryption_properties(
kms_connection_config, encryption_config)
```

```
with pq.ParquetWriter(path,
table.schema,
encryption_properties=file_encryption_properties,
use_dictionary=False,
compression='none',) as writer:
writer.write_table(table)

```

This is the error that I keep getting

```
File "/Users/sayalaso/aidata/dataflow/VaultClient.py", line 149, in parquet_write_read_with_vault
file_encryption_properties = crypto_factory.file_encryption_properties(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_parquet_encryption.pyx", line 423, in pyarrow._parquet_encryption.CryptoFactory.file_encryption_properties
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Failed to get key material store.
The Parquet file path must be specified when using external key material
```

although as per the Doc it says that: `If set to false, key material is stored in separate files in the same folder`

Using: pyarrow~=12.0.0

### Component(s)

Parquet, Python, PyArrow

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the example with Python 12.0.0, focusing on EncryptionConfiguration.internal_key_material and CryptoFactory.file_encryption_properties. Trace how ParquetWriter receives the encryption properties and compare the observed path requirement with the documented behavior; done means the external-key-material behavior is clarified through a fix or documentation update.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.