apache / apache/gluten

[VL] Native Parquet writer ignores the parquet.compression table property

Open
#12,987 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

With `spark.sql.hive.convertMetastoreParquet=false`, the native writer ignores the `parquet.compression` table property and uses `spark.sql.parquet.compression.codec` instead.

Reproduce with a Hive table whose table property asks for zstd while the session conf asks for gzip:

```sql
SET spark.sql.hive.convertMetastoreParquet=false;
SET spark.sql.parquet.compression.codec=gzip;
CREATE TABLE t STORED AS PARQUET TBLPROPERTIES ('parquet.compression'='zstd') AS SELECT 1 as c;
```

The Parquet footer reports GZIP. Vanilla Spark honours the table property here, so the written file differs depending on whether the native writer is on.

### How this surfaced

`VeloxParquetWriteForHiveSuite."native writer should respect table properties"` already compared the codec against zstd, but the comparison was the last expression of a `Utils.tryWithResource` block and its value was discarded, so the case passed regardless. #12981 wrapped it in `assert`, and it fails on 3.4, 3.5, 4.0 and 4.1:

```
- native writer should respect table properties *** FAILED ***
"zstd".equalsIgnoreCase(column.getCodec().toString()) was false
expected zstd but got GZIP in file:.../VeloxParquetWriteForHiveSuite/t/part-00000-...,
nativeWriteEnabled=true (VeloxParquetWriteForHiveSuite.scala:294)
```

The case is `ignore`d in #12981 until this is fixed, with a comment pointing here.

One thing the test cannot tell us yet: the case loops over `Seq(true, false)` for `enableNativeWrite`, and the assertion aborts on the first iteration, so the vanilla writer path was never reached. Whether vanilla also writes GZIP under these settings is unverified.

### Spark version

None

### Spark configurations

`spark.sql.hive.convertMetastoreParquet=false`, `spark.sql.parquet.compression.codec=gzip`, `spark.gluten.sql.native.writer.enabled=true`

### System information

N/A

### Relevant logs

```
- native writer should respect table properties *** FAILED ***
"zstd".equalsIgnoreCase(column.getCodec().toString()) was false
expected zstd but got GZIP in file:/__w/gluten/gluten/backends-velox/spark-warehouse/org.apache.spark.sql.execution.VeloxParquetWriteForHiveSuite/t/part-00000-8619fbf4-c5f2-419f-8a61-1c9a19a9b3ca,
nativeWriteEnabled=true (VeloxParquetWriteForHiveSuite.scala:294)
```

Contributor guide

Open the contributing guide

Research direction

Start with VeloxParquetWriteForHiveSuite.scala, especially the "native writer should respect table properties" case around line 294, and run it with the listed session and table-property settings. Trace how the native writer selects compression, then restore the assertion and verify that the native path writes zstd while the vanilla path is also checked.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, databases, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.