apache / apache/datafusion-comet
rlike: consider arrow regexp_is_match kernel and fix non-StringArray panic
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
`RLike::is_match` in `native/spark-expr/src/predicate_funcs/rlike.rs` loops over a `StringArray` calling `regex::Regex::is_match` per row into a `BooleanBuilder`. Arrow's `regexp_is_match` kernel does the same thing: it is built on the same Rust `regex` crate, so the documented divergence from Java regex semantics is unchanged, and null propagation matches.
The kernel would also make the array path more robust: it supports Utf8/LargeUtf8/Utf8View, while the current code hard-`expect`s a `StringArray` and would panic on other string layouts.
### Describe the potential solution
Replace the row loop with `regexp_is_match(array, &StringArray::new_scalar(pattern), None)`.
Trade-offs to resolve:
- The struct currently pre-compiles the `Regex` once at plan build; the kernel takes a pattern string and compiles per batch. For complex patterns this is a measurable regression; there is no arrow API accepting a pre-built `Regex`. Either accept per-batch compilation or keep the loop and only fix the downcast robustness.
- The dictionary fast path (run on dictionary values then `take`) and the scalar-input branch stay as wrappers.
### Additional context
Found during an audit of native code that replicates existing arrow-rs kernels.
Contributor guide
Assessment
This issue has not been assessed yet.