apache / apache/datafusion

Invalid parquet writer options (e.g. `max_row_group_size = 0`, `bloom_filter_fpp = 1.5`) panic in `COPY` instead of returning an error

Open
#24,904 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`ParquetOptions::into_writer_properties_builder` (`datafusion/common/src/file_options/parquet_writer.rs`) passes the session options straight to the `parquet` crate's `WriterPropertiesBuilder`. Several of its setters `assert!` on values they cannot accept, so an invalid `SET` turns the next parquet write into a panic (or a hang) instead of a configuration error.

### To Reproduce

```sql
SET datafusion.execution.parquet.max_row_group_size = 0;
COPY (SELECT 1) TO 'x.parquet';
```

```text
thread 'main' panicked at parquet-59.2.0/src/file/properties.rs:756:9:
assertion `left != right` failed: Cannot have a 0 max row group row count
```

```sql
SET datafusion.execution.parquet.bloom_filter_fpp = 1.5; -- also 0, 1, -1, NaN
COPY (SELECT 1) TO 'x.parquet';
```

```text
thread 'main' panicked at parquet-59.2.0/src/file/properties.rs:1665:13:
bloom filter fpp must be between 0 and 1 (exclusive) ...
```

```sql
SET datafusion.execution.parquet.write_batch_size = 0;
COPY (SELECT 1) TO 'x.parquet'; -- never returns
```

The same applies to `column_index_truncate_length = 0`, `statistics_truncate_length = 0`, a per-column `bloom_filter_fpp`, and `content_defined_chunking.min_chunk_size = 0` / `max_chunk_size <= min_chunk_size`.

### Expected behavior

A configuration error naming the option, like the existing handling of an unknown `compression` or `encoding` string in the same function.

### Additional context

Found while running a corpus of extreme configuration values against a debug build of `datafusion-cli`.

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/common/src/file_options/parquet_writer.rs at ParquetOptions::into_writer_properties_builder, then run the listed SET and COPY reproductions in datafusion-cli. Validate the affected options before parquet writer setters can panic or hang, returning an option-specific configuration error like the existing compression and encoding handling. Done means each invalid value fails with an error rather than panicking or hanging.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.