apache / apache/datafusion

`COPY ... PARTITIONED BY` a `Date32` / `Date64` column panics on out-of-range values

Offen
#24,907 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Rust
Sterne
9.3k
Forks
2.4k
Ø Merge
3 T. 11 Std.
Gemergte PRs (30 T.)
360

Beschreibung

### Describe the bug

`compute_partition_keys_by_row` in `datafusion/datasource/src/write/demux.rs` renders `Date32` and `Date64` partition values as `yyyy-mm-dd` path segments with unchecked arithmetic and an `unwrap`:

```rust
NaiveDate::from_num_days_from_ce_opt(EPOCH_DAYS_FROM_CE + array.value(i)).unwrap()
NaiveDate::from_num_days_from_ce_opt(EPOCH_DAYS_FROM_CE + (array.value(i) / 86_400_000) as i32).unwrap()
```

Any value that does not fit a `chrono::NaiveDate` panics the writer task.

### To Reproduce

```sql
COPY (VALUES (arrow_cast(2147483647, 'Date32'))) TO 'out1/' STORED AS parquet PARTITIONED BY (column1);
-- thread 'tokio-rt-worker' panicked at datafusion/datasource/src/write/demux.rs:427:25:
-- attempt to add with overflow

COPY (VALUES (arrow_cast(-2147483648, 'Date32'))) TO 'out2/' STORED AS parquet PARTITIONED BY (column1);
-- demux.rs:429:22: called `Option::unwrap()` on a `None` value

COPY (VALUES (arrow_cast(9223372036854775807, 'Date64'))) TO 'out3/' STORED AS parquet PARTITIONED BY (column1);
-- demux.rs:443:22: called `Option::unwrap()` on a `None` value
```

### Expected behavior

An execution error naming the value that cannot be rendered as a date, instead of a panic.

### Additional context

The `Date64` branch also truncates toward zero (`/ 86_400_000`), so a value shortly before the epoch (e.g. `-1` ms) is placed in the `1970-01-01` partition rather than `1969-12-31`.

Found while running a corpus of extreme-value literals against a debug build of `datafusion-cli`.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start in datafusion/datasource/src/write/demux.rs at compute_partition_keys_by_row and reproduce the listed COPY statements with datafusion-cli. Trace the Date32 and Date64 rendering branches, including the negative-millisecond case. Done means out-of-range values return an execution error naming the value instead of panicking, and values before the epoch use the correct date partition.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
rust, sql
Bereich
databases
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Aktiv
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
72/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.