Spark procedure filter evaluator drops all rows for any array predicate (array values not wrapped as ArrayData)
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem**
`HoodieProcedureFilterUtils.convertValueToInternal` converts array columns (`Seq` / `Array` / `java.util.List`) to a raw `Object[]` instead of Catalyst `ArrayData`. Evaluating any predicate touching an array column -- `size(...)`, `array_contains(...)`, even `isnotnull(col)` -- throws `ClassCastException` inside `BoundReference.eval`, which the per-row `Try` swallows to `false`, so every row is dropped. The `sort_array` / `array_size` entries in the function-resolution table are dead code. (Maps are unaffected: they go through `ArrayBasedMapData`.)
**To reproduce**
Pinned in `TestHoodieProcedureFilterUtils` ("converts array / decimal / binary / uuid / java-time columns") since #19161: `size(arrScala) >= 0` and `isnotnull(arrScala)` both return no rows.
**Suggested fix**
Wrap array values in `new GenericArrayData(...)` in `convertValueToInternal`, then flip the pinned assertions.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.