apache / apache/datafusion-comet

Add Java-generated Rust parity fixtures for regex crate upgrades

Open
#5,813 0 comments 0 reactions 0 assignees View on GitHub
enhancement requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

The `CometRegex` whitelist introduced by PR #5415 depends on the semantics and compilation behavior of the locked Rust `regex` crate.

The existing `CometRegexParitySuite` exercises the native kernel, but its admitted pattern corpus is relatively small. A future `regex` crate upgrade could change the result or compilation status of an admitted pattern without the current corpus detecting it.

This creates coupling between the plan-time admission rules and the specific `regex` crate version.

### Describe the potential solution

Generate a committed table of `(pattern, subject, expected)` fixtures using Java's `Pattern.matcher(subject).find()` and execute those fixtures directly in the Rust `rlike` unit tests.

The fixtures should cover:

- every construct currently admitted by `CometRegex`;
- concatenation and alternation combinations;
- capturing and non-capturing groups;
- character classes, ranges, negated classes, and escaped literals;
- greedy and counted quantifiers;
- empty-match cases;
- ASCII, non-ASCII, control-character, newline, and supplementary-code-point subjects;
- representative patterns near the conservative structural limits.

Document the JDK version and fixture-generation process. The committed Rust test should not require a JVM and should report the pattern, subject, expected result, and actual result on failure.

A Rust `regex` crate upgrade that changes the semantics or compilation status of an admitted fixture should cause this test to fail until the change is deliberately evaluated.

### Additional context

Follow-up from #5415 and #5351.

This tracks the Rust-side Java-generated fixture table requested during review of #5415: https://github.com/apache/datafusion-comet/pull/5415#pullrequestreview-5154502402

The review specifically highlighted the regex crate-version coupling.

Contributor guide

Open the contributing guide

Research direction

Start with CometRegexParitySuite and the Rust rlike unit tests, then review PR #5415 and issue #5351 for the admission rules and review context. Done means committed Java-generated fixtures cover the listed constructs and subjects, Rust tests run without a JVM, failures show pattern, subject, expected, and actual values, and the JDK and generation process are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, rust
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.