[Python][Parquet][PyArrow] Cannot save footer files outside the parquet file
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the usage question you have. Please include as many useful details as possible.
Hi Team,
I have a use case where I'm trying to encrypt my parquet files and save the keys externally so it's easier to rotate the master keys and re-encrypt the DEK without reading the original file.
However, I'm unable to save the footer files separately with just specifying the `internal_key_material=False` in `EncryptionConfiguration`. Is there something else missing in the documentation or is this is a bug ?
Here's what I'm doing:
```
encryption_config = pe.EncryptionConfiguration(
footer_key=key_name,
column_keys={
key_name: ["a", "b"],
},
plaintext_footer=True,
internal_key_material=False,
data_key_length_bits=256,
)
```
```
file_encryption_properties = crypto_factory.file_encryption_properties(
kms_connection_config, encryption_config)
```
```
with pq.ParquetWriter(path,
table.schema,
encryption_properties=file_encryption_properties,
use_dictionary=False,
compression='none',) as writer:
writer.write_table(table)
```
This is the error that I keep getting
```
File "/Users/sayalaso/aidata/dataflow/VaultClient.py", line 149, in parquet_write_read_with_vault
file_encryption_properties = crypto_factory.file_encryption_properties(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_parquet_encryption.pyx", line 423, in pyarrow._parquet_encryption.CryptoFactory.file_encryption_properties
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Failed to get key material store.
The Parquet file path must be specified when using external key material
```
although as per the Doc it says that: `If set to false, key material is stored in separate files in the same folder`
Using: pyarrow~=12.0.0
### Component(s)
Parquet, Python, PyArrow
Contributor guide
Research direction
Start by reproducing the example with Python 12.0.0, focusing on EncryptionConfiguration.internal_key_material and CryptoFactory.file_encryption_properties. Trace how ParquetWriter receives the encryption properties and compare the observed path requirement with the documented behavior; done means the external-key-material behavior is clarified through a fix or documentation update.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100