Expand procedure filter's supported functions via Spark FunctionRegistry fallback
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Context
Follow-up to #19638 / #19850, as discussed with @voonhous [here](https://github.com/apache/hudi/issues/19638#issuecomment-5559550232).
#19850 fixes procedure filters so unsupported functions are rejected with a clear error at validation time, instead of silently returning zero rows. That's the correct safety fix.
This issue proposes a complementary enhancement: expand the set of functions `HoodieProcedureFilterUtils` actually supports, by falling back to Spark's own `FunctionRegistry` to resolve any function not covered by the hand-maintained resolution table, instead of rejecting it. This would let filters use any Spark builtin (e.g. `concat`, `instr`) rather than only the ones explicitly enumerated.
### Proposed approach
- In the shared bind/resolve pipeline, when a function isn't found in the existing hardcoded table, attempt `sparkSession.sessionState.functionRegistry.lookupFunction(FunctionIdentifier(name), args)`.
- If that also fails to resolve (e.g. a genuine typo), fall through to the existing validation-rejection behavior from #19850 — no change to safety guarantees for truly unknown names.
- Needs compatibility tests per component/Spark version, since the resolvable function set now depends on Spark's registry contents rather than a fixed list.
### Blocked on
- #19638 (needs #19850 merged first, since the fallback builds on the shared resolver introduced there)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.