`COPY ... PARTITIONED BY` a `Date32` / `Date64` column panics on out-of-range values
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
`compute_partition_keys_by_row` in `datafusion/datasource/src/write/demux.rs` renders `Date32` and `Date64` partition values as `yyyy-mm-dd` path segments with unchecked arithmetic and an `unwrap`:
```rust
NaiveDate::from_num_days_from_ce_opt(EPOCH_DAYS_FROM_CE + array.value(i)).unwrap()
NaiveDate::from_num_days_from_ce_opt(EPOCH_DAYS_FROM_CE + (array.value(i) / 86_400_000) as i32).unwrap()
```
Any value that does not fit a `chrono::NaiveDate` panics the writer task.
### To Reproduce
```sql
COPY (VALUES (arrow_cast(2147483647, 'Date32'))) TO 'out1/' STORED AS parquet PARTITIONED BY (column1);
-- thread 'tokio-rt-worker' panicked at datafusion/datasource/src/write/demux.rs:427:25:
-- attempt to add with overflow
COPY (VALUES (arrow_cast(-2147483648, 'Date32'))) TO 'out2/' STORED AS parquet PARTITIONED BY (column1);
-- demux.rs:429:22: called `Option::unwrap()` on a `None` value
COPY (VALUES (arrow_cast(9223372036854775807, 'Date64'))) TO 'out3/' STORED AS parquet PARTITIONED BY (column1);
-- demux.rs:443:22: called `Option::unwrap()` on a `None` value
```
### Expected behavior
An execution error naming the value that cannot be rendered as a date, instead of a panic.
### Additional context
The `Date64` branch also truncates toward zero (`/ 86_400_000`), so a value shortly before the epoch (e.g. `-1` ms) is placed in the `1970-01-01` partition rather than `1969-12-31`.
Found while running a corpus of extreme-value literals against a debug build of `datafusion-cli`.
Contributor guide
Research direction
Start in datafusion/datasource/src/write/demux.rs at compute_partition_keys_by_row and reproduce the listed COPY statements with datafusion-cli. Trace the Date32 and Date64 rendering branches, including the negative-millisecond case. Done means out-of-range values return an execution error naming the value instead of panicking, and values before the epoch use the correct date partition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100