apache / apache/datafusion-comet

arrays_overlap panics on nested double arrays built from expressions

Open
#5,269 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions array expressions bug crash priority:high
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Describe the bug

`arrays_overlap` over `array>` panics in native code when both sides of the comparison are expression-constructed nested lists rather than plain columns.

```
org.apache.comet.CometNativeException: native panic: index out of bounds: the len is 2 but the index is 2
Comet native panic: panicked at arrow-array-58.4.0/src/array/list_array.rs:352:19
```

`list_array.rs:352` is `GenericListArray::value`, so something in `arrays_overlap_list_generic` (`native/spark-expr/src/array_funcs/arrays_overlap.rs`) indexes past the end of one of its two list arrays.

## Steps to reproduce

As a Comet SQL file test:

```sql
statement
CREATE TABLE t_panic(a array>) USING parquet

statement
INSERT INTO t_panic VALUES
(array(array(-0.0))),
(array(array(0.0))),
(array(array(1.0, -0.0))),
(array(array(double('NaN')))),
(array(array(1.0))),
(array(array(-0.0), cast(NULL as array))),
(array(cast(NULL as array)))

query
SELECT arrays_overlap(array(array(a[0][0])), array(array(cast(0.0 as double)))) FROM t_panic
```

Reproduced on `apache/main` at `f262b13d2`, Spark 4.1 profile, JDK 17, macOS.

## What I could and could not narrow down

The panic needs all seven rows. None of these reproduce it on their own:

- any single row from the table, including the two rows carrying a `NULL` inner list
- a two-row table combining a multi-element inner list with a `NULL` inner list
- the `int` equivalent of the same query
- `arrays_overlap(a, array(array(cast(0.0 as double))))`, a plain column against a constant
- `arrays_overlap(array(a[0]), array(array(0)))`, only the outer list constructed

The failing task index moves with the number of rows, so this looks like a batch or partition boundary interacting with the per-row `left.value(i)` / `right.value(i)` slicing in `arrays_overlap_list_generic`, rather than anything about a specific row's contents. Worth checking whether the two sides can end up with different `len()` when one of them arrives as a broadcast scalar and the other is a per-row constructed array.

## Additional context

Found while reviewing #5235, which changes the same function. The panic reproduces identically with and without that PR applied, so it is independent of it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.