apache / apache/parquet-java

parquet.encryption.plaintext.footer parameter being true, code expects parquet.encryption.footer.key

Open
#2,716 1 comment 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Hi Team,

I want my footer in parquet file to be non encrypted. so I set the _parquet.encryption.plaintext.footer_ to be {_}true{_}, but when I tried to run my code, parquet-mr is expecting __ value __ for the __ property _parquet.encryption.footer.key  \*\*_  

Reproducer

Spark 3.3.0 

Download the [file\|[https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.12.0/parquet-hadoop-1.12.0-tests.jar] ] and place it in spark - jar directory 

using spark-shell
```java

sc.hadoopConfiguration.set("parquet.crypto.factory.class" ,"org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory") sc.hadoopConfiguration.set("parquet.encryption.kms.client.class" ,"org.apache.parquet.crypto.keytools.mocks.InMemoryKMS") sc.hadoopConfiguration.set("parquet.encryption.key.list","key1a: BAECAwQFBgcICQoLDA0ODw==, key2a: BAECAAECAAECAAECAAECAA==, keyz: BAECAAECAAECAAECAAECAA==") sc.hadoopConfiguration.set("parquet.encryption.key.material.store.internally","false")
val encryptedParquetPath = "/tmp/par_enc_footer_non_encrypted"
val partitionCol = 1
case class nestedItem(ic: Int = 0, sic : Double, pc: Int = 0)
case class SquareItem(int_column: Int, square_int_column : Double, partitionCol: Int, nestedCol :nestedItem)
val dataRange = (1 to 100).toList
val squares = sc.parallelize(dataRange.map(i => new SquareItem(i, scala.math.pow(i,2), partitionCol,nestedItem(i,i))))
squares.toDS().show() squares.toDS().write.partitionBy("partitionCol").mode("overwrite").option("parquet.encryption.column.keys", "key1a:square_int_column,nestedCol.ic;").option("parquet.encryption.plaintext.footer",true).parquet(encryptedParquetPath)
```
I get the below error, my expectation is if I set properties for my footer to be plain text, why do we need keys for footer.

 
```java

 
Caused by: org.apache.parquet.crypto.ParquetCryptoRuntimeException: Undefined footer key
  at org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory.getFileEncryptionProperties(PropertiesDrivenCryptoFactory.java:88)
  at org.apache.parquet.hadoop.ParquetOutputFormat.createEncryptionProperties(ParquetOutputFormat.java:554)
  at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:478)
  at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:420)
  at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:409)
  at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.(ParquetOutputWriter.scala:36)
  at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:155)
  at org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.renewCurrentWriter(FileFormatDataWriter.scala:298)
  at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:365)
  at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
  at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:331)
  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:338)
  ... 9 more
```
 

**Reporter**: [Vignesh Nageswaran](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Nageswaran)

**Note**: *This issue was originally created as [PARQUET-2194](https://issues.apache.org/jira/browse/PARQUET-2194). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reproducer and the stack trace at PropertiesDrivenCryptoFactory.java:88, then inspect ParquetOutputFormat.java around encryption-property creation. Verify whether parquet.encryption.plaintext.footer=true still requires parquet.encryption.footer.key, and define done as the reported configuration completing without the Undefined footer key error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.