apache / apache/hudi

Expand procedure filter's supported functions via Spark FunctionRegistry fallback

Open
#19,852 1 comment 0 reactions 1 assignee Claimed by @anshulsingh-py View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Context
Follow-up to #19638 / #19850, as discussed with @voonhous [here](https://github.com/apache/hudi/issues/19638#issuecomment-5559550232).

#19850 fixes procedure filters so unsupported functions are rejected with a clear error at validation time, instead of silently returning zero rows. That's the correct safety fix.

This issue proposes a complementary enhancement: expand the set of functions `HoodieProcedureFilterUtils` actually supports, by falling back to Spark's own `FunctionRegistry` to resolve any function not covered by the hand-maintained resolution table, instead of rejecting it. This would let filters use any Spark builtin (e.g. `concat`, `instr`) rather than only the ones explicitly enumerated.

### Proposed approach
- In the shared bind/resolve pipeline, when a function isn't found in the existing hardcoded table, attempt `sparkSession.sessionState.functionRegistry.lookupFunction(FunctionIdentifier(name), args)`.
- If that also fails to resolve (e.g. a genuine typo), fall through to the existing validation-rejection behavior from #19850 — no change to safety guarantees for truly unknown names.
- Needs compatibility tests per component/Spark version, since the resolvable function set now depends on Spark's registry contents rather than a fixed list.

### Blocked on
- #19638 (needs #19850 merged first, since the fallback builds on the shared resolver introduced there)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.