apache / apache/datafusion-comet
arrays_overlap panics on nested double arrays built from expressions
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Describe the bug
`arrays_overlap` over `array>` panics in native code when both sides of the comparison are expression-constructed nested lists rather than plain columns.
```
org.apache.comet.CometNativeException: native panic: index out of bounds: the len is 2 but the index is 2
Comet native panic: panicked at arrow-array-58.4.0/src/array/list_array.rs:352:19
```
`list_array.rs:352` is `GenericListArray::value`, so something in `arrays_overlap_list_generic` (`native/spark-expr/src/array_funcs/arrays_overlap.rs`) indexes past the end of one of its two list arrays.
## Steps to reproduce
As a Comet SQL file test:
```sql
statement
CREATE TABLE t_panic(a array>) USING parquet
statement
INSERT INTO t_panic VALUES
(array(array(-0.0))),
(array(array(0.0))),
(array(array(1.0, -0.0))),
(array(array(double('NaN')))),
(array(array(1.0))),
(array(array(-0.0), cast(NULL as array))),
(array(cast(NULL as array)))
query
SELECT arrays_overlap(array(array(a[0][0])), array(array(cast(0.0 as double)))) FROM t_panic
```
Reproduced on `apache/main` at `f262b13d2`, Spark 4.1 profile, JDK 17, macOS.
## What I could and could not narrow down
The panic needs all seven rows. None of these reproduce it on their own:
- any single row from the table, including the two rows carrying a `NULL` inner list
- a two-row table combining a multi-element inner list with a `NULL` inner list
- the `int` equivalent of the same query
- `arrays_overlap(a, array(array(cast(0.0 as double))))`, a plain column against a constant
- `arrays_overlap(array(a[0]), array(array(0)))`, only the outer list constructed
The failing task index moves with the number of rows, so this looks like a batch or partition boundary interacting with the per-row `left.value(i)` / `right.value(i)` slicing in `arrays_overlap_list_generic`, rather than anything about a specific row's contents. Worth checking whether the two sides can end up with different `len()` when one of them arrives as a broadcast scalar and the other is a per-row constructed array.
## Additional context
Found while reviewing #5235, which changes the same function. The panic reproduces identically with and without that PR applied, so it is independent of it.
Contributor guide
Assessment
This issue has not been assessed yet.