apache / apache/hudi

Spark procedure filter evaluator drops all rows for any array predicate (array values not wrapped as ArrayData)

Open
#19,633 2 comments 0 reactions 1 assignee Claimed by @LeonxLJX View on GitHub
good first issue
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem**

`HoodieProcedureFilterUtils.convertValueToInternal` converts array columns (`Seq` / `Array` / `java.util.List`) to a raw `Object[]` instead of Catalyst `ArrayData`. Evaluating any predicate touching an array column -- `size(...)`, `array_contains(...)`, even `isnotnull(col)` -- throws `ClassCastException` inside `BoundReference.eval`, which the per-row `Try` swallows to `false`, so every row is dropped. The `sort_array` / `array_size` entries in the function-resolution table are dead code. (Maps are unaffected: they go through `ArrayBasedMapData`.)

**To reproduce**

Pinned in `TestHoodieProcedureFilterUtils` ("converts array / decimal / binary / uuid / java-time columns") since #19161: `size(arrScala) >= 0` and `isnotnull(arrScala)` both return no rows.

**Suggested fix**

Wrap array values in `new GenericArrayData(...)` in `convertValueToInternal`, then flip the pinned assertions.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.