apache / apache/datafusion-comet

Native Parquet writer ignores Spark's Parquet writer properties (block size, page size, dictionary, writer version, statistics)

Open
#5,304 0 comments 0 reactions 0 assignees View on GitHub
area:writer enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

`ParquetWriterExec` builds its `WriterProperties` with compression and nothing else (`native/core/src/execution/operators/parquet_writer.rs`):

```rust
let props = WriterProperties::builder()
.set_compression(compression)
.build();
```

Every other Parquet write knob Spark exposes is therefore silently ignored, and files come out with arrow-rs defaults rather than the ones the user configured:

- `parquet.block.size` (row group size) — tracked as an unchecked item on #2967
- `parquet.page.size` / `parquet.page.row.count.limit`
- `parquet.enable.dictionary` and dictionary page size
- `spark.sql.parquet.writer.version` (`PARQUET_1_0` / `PARQUET_2_0`)
- statistics level, and bloom filter settings
- `spark.sql.files.maxRecordsPerFile` (needs file rolling, so partly separate)

Row group size is the one with real consequences: the default row group size decides read parallelism and memory for every downstream consumer, and a user who set `parquet.block.size` gets no error, just a differently shaped file.

This also matters beyond the V1 write path — #4658 splits Iceberg writes into writer + committer operators with the stated goal of writing data files natively next, and Iceberg data files are read by engines other than Spark.

### Describe the potential solution

Thread the resolved options through the `ParquetWriter` protobuf message and set them on `WriterProperties`. Spark's `ParquetOptions` / `ParquetWriteSupport` are the reference for precedence and defaults.

Related: #2814 (compression settings), #3425 (INT96), #3427 (Spark version in footer metadata).

Contributor guide

Open the contributing guide

Research direction

Start in native/core/src/execution/operators/parquet_writer.rs and locate the ParquetWriter protobuf message used by ParquetWriterExec. Read Spark's ParquetOptions and ParquetWriteSupport for precedence and defaults, then verify that the listed writer options reach WriterProperties; completion should cover the stated properties without changing the separately noted file-rolling work.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.