Rework/Reimagine/Cleanup Regex Kernels
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
The current `regexp_is_match_utf8` and `regexp_is_match_utf8_scalar` kernels could stand to be improved:
* Predate the Datum abstraction making them clunky to use
* Make use of low-level ArrayData instead of newer more ergonomic abstractions
* No story for accessing capture byte offsets - https://github.com/apache/arrow-rs/issues/5966
* No story for advanced regex configuration - https://github.com/apache/arrow-rs/issues/5989
* No support for StringViewArray
* No support for binary regex matching
I can see a couple of paths forward here:
1. We improve the expressiveness of the existing kernels, adding more kernels and configuration options
2. We simplify the implementation of the kernels and better document how people can implement their own kernels to process regex
Given the API surface of the regex crate, and how simple it actually is to write a regex kernel, I am personally leaning towards 2.
Contributor guide
Research direction
Start by reviewing the regexp_is_match_utf8 and regexp_is_match_utf8_scalar kernels, then read linked issues 5966 and 5989 for the capture-offset and regex-configuration requirements. Decide between expanding the existing kernel API and simplifying the implementation with guidance for custom regex kernels; done requires a settled direction that addresses the listed limitations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100