apache / apache/datafusion

`COPY ... PARTITIONED BY` a `Date32` / `Date64` column panics on out-of-range values

Open
#24,907 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`compute_partition_keys_by_row` in `datafusion/datasource/src/write/demux.rs` renders `Date32` and `Date64` partition values as `yyyy-mm-dd` path segments with unchecked arithmetic and an `unwrap`:

```rust
NaiveDate::from_num_days_from_ce_opt(EPOCH_DAYS_FROM_CE + array.value(i)).unwrap()
NaiveDate::from_num_days_from_ce_opt(EPOCH_DAYS_FROM_CE + (array.value(i) / 86_400_000) as i32).unwrap()
```

Any value that does not fit a `chrono::NaiveDate` panics the writer task.

### To Reproduce

```sql
COPY (VALUES (arrow_cast(2147483647, 'Date32'))) TO 'out1/' STORED AS parquet PARTITIONED BY (column1);
-- thread 'tokio-rt-worker' panicked at datafusion/datasource/src/write/demux.rs:427:25:
-- attempt to add with overflow

COPY (VALUES (arrow_cast(-2147483648, 'Date32'))) TO 'out2/' STORED AS parquet PARTITIONED BY (column1);
-- demux.rs:429:22: called `Option::unwrap()` on a `None` value

COPY (VALUES (arrow_cast(9223372036854775807, 'Date64'))) TO 'out3/' STORED AS parquet PARTITIONED BY (column1);
-- demux.rs:443:22: called `Option::unwrap()` on a `None` value
```

### Expected behavior

An execution error naming the value that cannot be rendered as a date, instead of a panic.

### Additional context

The `Date64` branch also truncates toward zero (`/ 86_400_000`), so a value shortly before the epoch (e.g. `-1` ms) is placed in the `1970-01-01` partition rather than `1969-12-31`.

Found while running a corpus of extreme-value literals against a debug build of `datafusion-cli`.

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/datasource/src/write/demux.rs at compute_partition_keys_by_row and reproduce the listed COPY statements with datafusion-cli. Trace the Date32 and Date64 rendering branches, including the negative-millisecond case. Done means out-of-range values return an execution error naming the value instead of panicking, and values before the epoch use the correct date partition.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.