apache / apache/datafusion-comet
Measure CometRegex whitelist coverage on representative regex patterns
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
The initial `CometRegex` whitelist introduced by PR #5415 is deliberately conservative. It rejects commonly used constructs such as `.`, `^`, `$`, `\d`, `\w`, and `\s`.
We currently do not know what fraction of representative real-world `rlike` patterns is admitted for automatic native execution. TPC-DS does not contain useful `rlike` coverage, so it cannot answer this question.
Without an admission-rate measurement, it is difficult to prioritize follow-up compatibility work or estimate how much real-world workload coverage the whitelist provides.
### Describe the potential solution
Identify one or more representative regex corpora, such as public SQL workloads, open-source application queries, or safely shareable production-derived patterns.
Run every literal pattern through `CometRegex` and report:
- the corpus source and collection method;
- total, admitted, and rejected pattern counts;
- the admission percentage;
- duplicate-handling methodology;
- rejection counts grouped by unsupported construct or analyzer rule;
- useful pattern-length and complexity distributions.
Use the results to prioritize follow-up work such as support for leading `^` or safe normalization of selected Java regex constructs.
The whitelist should not be weakened solely to increase the reported admission percentage.
### Additional context
Follow-up from #5415 and #5351.
This tracks the representative admission-rate measurement requested during review of #5415: https://github.com/apache/datafusion-comet/pull/5415#pullrequestreview-5154502402
The original issue notes that the TPC query corpus does not contain `rlike` usage: https://github.com/apache/datafusion-comet/issues/5351
Contributor guide
Research direction
Start by locating CometRegex and the handling of rlike, then review PR #5415 and issue #5351 for whitelist behavior and context. Identify a representative corpus and define collection, deduplication, admission, rejection grouping, and complexity measurements; done means a reproducible report covering the requested counts and distributions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- data, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100