[VL] Native Parquet writer ignores the parquet.compression table property
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
With `spark.sql.hive.convertMetastoreParquet=false`, the native writer ignores the `parquet.compression` table property and uses `spark.sql.parquet.compression.codec` instead.
Reproduce with a Hive table whose table property asks for zstd while the session conf asks for gzip:
```sql
SET spark.sql.hive.convertMetastoreParquet=false;
SET spark.sql.parquet.compression.codec=gzip;
CREATE TABLE t STORED AS PARQUET TBLPROPERTIES ('parquet.compression'='zstd') AS SELECT 1 as c;
```
The Parquet footer reports GZIP. Vanilla Spark honours the table property here, so the written file differs depending on whether the native writer is on.
### How this surfaced
`VeloxParquetWriteForHiveSuite."native writer should respect table properties"` already compared the codec against zstd, but the comparison was the last expression of a `Utils.tryWithResource` block and its value was discarded, so the case passed regardless. #12981 wrapped it in `assert`, and it fails on 3.4, 3.5, 4.0 and 4.1:
```
- native writer should respect table properties *** FAILED ***
"zstd".equalsIgnoreCase(column.getCodec().toString()) was false
expected zstd but got GZIP in file:.../VeloxParquetWriteForHiveSuite/t/part-00000-...,
nativeWriteEnabled=true (VeloxParquetWriteForHiveSuite.scala:294)
```
The case is `ignore`d in #12981 until this is fixed, with a comment pointing here.
One thing the test cannot tell us yet: the case loops over `Seq(true, false)` for `enableNativeWrite`, and the assertion aborts on the first iteration, so the vanilla writer path was never reached. Whether vanilla also writes GZIP under these settings is unverified.
### Spark version
None
### Spark configurations
`spark.sql.hive.convertMetastoreParquet=false`, `spark.sql.parquet.compression.codec=gzip`, `spark.gluten.sql.native.writer.enabled=true`
### System information
N/A
### Relevant logs
```
- native writer should respect table properties *** FAILED ***
"zstd".equalsIgnoreCase(column.getCodec().toString()) was false
expected zstd but got GZIP in file:/__w/gluten/gluten/backends-velox/spark-warehouse/org.apache.spark.sql.execution.VeloxParquetWriteForHiveSuite/t/part-00000-8619fbf4-c5f2-419f-8a61-1c9a19a9b3ca,
nativeWriteEnabled=true (VeloxParquetWriteForHiveSuite.scala:294)
```
Contributor guide
Research direction
Start with VeloxParquetWriteForHiveSuite.scala, especially the "native writer should respect table properties" case around line 294, and run it with the listed session and table-property settings. Trace how the native writer selects compression, then restore the assertion and verify that the native path writes zstd while the vanilla path is also checked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, databases, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100