apache / apache/arrow-rs

Rework/Reimagine/Cleanup Regex Kernels

Open
#5,991 0 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

The current `regexp_is_match_utf8` and `regexp_is_match_utf8_scalar` kernels could stand to be improved:

* Predate the Datum abstraction making them clunky to use
* Make use of low-level ArrayData instead of newer more ergonomic abstractions
* No story for accessing capture byte offsets - https://github.com/apache/arrow-rs/issues/5966
* No story for advanced regex configuration - https://github.com/apache/arrow-rs/issues/5989
* No support for StringViewArray
* No support for binary regex matching

I can see a couple of paths forward here:

1. We improve the expressiveness of the existing kernels, adding more kernels and configuration options
2. We simplify the implementation of the kernels and better document how people can implement their own kernels to process regex

Given the API surface of the regex crate, and how simple it actually is to write a regex kernel, I am personally leaning towards 2.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the regexp_is_match_utf8 and regexp_is_match_utf8_scalar kernels, then read linked issues 5966 and 5989 for the capture-offset and regex-configuration requirements. Decide between expanding the existing kernel API and simplifying the implementation with guidance for custom regex kernels; done requires a settled direction that addresses the listed limitations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.