Invalid parquet writer options (e.g. `max_row_group_size = 0`, `bloom_filter_fpp = 1.5`) panic in `COPY` instead of returning an error
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
`ParquetOptions::into_writer_properties_builder` (`datafusion/common/src/file_options/parquet_writer.rs`) passes the session options straight to the `parquet` crate's `WriterPropertiesBuilder`. Several of its setters `assert!` on values they cannot accept, so an invalid `SET` turns the next parquet write into a panic (or a hang) instead of a configuration error.
### To Reproduce
```sql
SET datafusion.execution.parquet.max_row_group_size = 0;
COPY (SELECT 1) TO 'x.parquet';
```
```text
thread 'main' panicked at parquet-59.2.0/src/file/properties.rs:756:9:
assertion `left != right` failed: Cannot have a 0 max row group row count
```
```sql
SET datafusion.execution.parquet.bloom_filter_fpp = 1.5; -- also 0, 1, -1, NaN
COPY (SELECT 1) TO 'x.parquet';
```
```text
thread 'main' panicked at parquet-59.2.0/src/file/properties.rs:1665:13:
bloom filter fpp must be between 0 and 1 (exclusive) ...
```
```sql
SET datafusion.execution.parquet.write_batch_size = 0;
COPY (SELECT 1) TO 'x.parquet'; -- never returns
```
The same applies to `column_index_truncate_length = 0`, `statistics_truncate_length = 0`, a per-column `bloom_filter_fpp`, and `content_defined_chunking.min_chunk_size = 0` / `max_chunk_size <= min_chunk_size`.
### Expected behavior
A configuration error naming the option, like the existing handling of an unknown `compression` or `encoding` string in the same function.
### Additional context
Found while running a corpus of extreme configuration values against a debug build of `datafusion-cli`.
Contributor guide
Research direction
Start in datafusion/common/src/file_options/parquet_writer.rs at ParquetOptions::into_writer_properties_builder, then run the listed SET and COPY reproductions in datafusion-cli. Validate the affected options before parquet writer setters can panic or hang, returning an option-specific configuration error like the existing compression and encoding handling. Done means each invalid value fails with an error rather than panicking or hanging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100