apache / apache/datafusion-comet
Native Parquet writer ignores Spark's Parquet writer properties (block size, page size, dictionary, writer version, statistics)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
`ParquetWriterExec` builds its `WriterProperties` with compression and nothing else (`native/core/src/execution/operators/parquet_writer.rs`):
```rust
let props = WriterProperties::builder()
.set_compression(compression)
.build();
```
Every other Parquet write knob Spark exposes is therefore silently ignored, and files come out with arrow-rs defaults rather than the ones the user configured:
- `parquet.block.size` (row group size) — tracked as an unchecked item on #2967
- `parquet.page.size` / `parquet.page.row.count.limit`
- `parquet.enable.dictionary` and dictionary page size
- `spark.sql.parquet.writer.version` (`PARQUET_1_0` / `PARQUET_2_0`)
- statistics level, and bloom filter settings
- `spark.sql.files.maxRecordsPerFile` (needs file rolling, so partly separate)
Row group size is the one with real consequences: the default row group size decides read parallelism and memory for every downstream consumer, and a user who set `parquet.block.size` gets no error, just a differently shaped file.
This also matters beyond the V1 write path — #4658 splits Iceberg writes into writer + committer operators with the stated goal of writing data files natively next, and Iceberg data files are read by engines other than Spark.
### Describe the potential solution
Thread the resolved options through the `ParquetWriter` protobuf message and set them on `WriterProperties`. Spark's `ParquetOptions` / `ParquetWriteSupport` are the reference for precedence and defaults.
Related: #2814 (compression settings), #3425 (INT96), #3427 (Spark version in footer metadata).
Contributor guide
Research direction
Start in native/core/src/execution/operators/parquet_writer.rs and locate the ParquetWriter protobuf message used by ParquetWriterExec. Read Spark's ParquetOptions and ParquetWriteSupport for precedence and defaults, then verify that the listed writer options reach WriterProperties; completion should cover the stated properties without changing the separately noted file-rolling work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100