Support stateful ScalarFunction
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Some of the `@ScalarFunction` we built and potentially other 3rd party customized function can benefit from a stateful optimization. E.g. if we know a scalarfunction is used with literal arguments. those literal arguments can be preprocessed (e.g. regexp pattern compilation)
However since ScalarFunction are used not only during the query code path but also the ingestion codepath. we need to be support both the stateful and stateless version of the function to ensure performance optimization and compatibility.
https://github.com/apache/pinot/pull/7443 created an example of how this could benefit. But it doesn't work on the ingestion path and there are some other issues to consider:
- [ ] need to guarantee that all the stateful version of the FunctionInvoker internal instances are properly initialized, in both query and ingestion code path
- [ ] support type matching with Literal detection, there's no point of optimizing scalar function if there's no prior knowledge of the arguments, might as well just do the computation during actual call.
- [ ] should not incur any additional branch condition in critical path.
- [ ] should ensure that multiple instances of the FunctionInvoker (and its underlying instances) doesn't share any states.
As a separate note. we also need to ensure that function signatures are matched to the argument type. and whether we expect the argument to be a literal or an expression. (i believe there's an issue open for this but cant find it yet)
Contributor guide
Research direction
Start by reviewing PR 7443 and the ScalarFunction and FunctionInvoker entry points involved in both query and ingestion paths. Trace how literal arguments and function signatures are matched, then define completion against the issue checklist: initialized stateful instances, isolated state, supported literal matching, and no added critical-path branching.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100