apache / apache/datafusion-comet

Measure CometRegex whitelist coverage on representative regex patterns

Open
#5,812 0 comments 0 reactions 0 assignees View on GitHub
enhancement requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

The initial `CometRegex` whitelist introduced by PR #5415 is deliberately conservative. It rejects commonly used constructs such as `.`, `^`, `$`, `\d`, `\w`, and `\s`.

We currently do not know what fraction of representative real-world `rlike` patterns is admitted for automatic native execution. TPC-DS does not contain useful `rlike` coverage, so it cannot answer this question.

Without an admission-rate measurement, it is difficult to prioritize follow-up compatibility work or estimate how much real-world workload coverage the whitelist provides.

### Describe the potential solution

Identify one or more representative regex corpora, such as public SQL workloads, open-source application queries, or safely shareable production-derived patterns.

Run every literal pattern through `CometRegex` and report:

- the corpus source and collection method;
- total, admitted, and rejected pattern counts;
- the admission percentage;
- duplicate-handling methodology;
- rejection counts grouped by unsupported construct or analyzer rule;
- useful pattern-length and complexity distributions.

Use the results to prioritize follow-up work such as support for leading `^` or safe normalization of selected Java regex constructs.

The whitelist should not be weakened solely to increase the reported admission percentage.

### Additional context

Follow-up from #5415 and #5351.

This tracks the representative admission-rate measurement requested during review of #5415: https://github.com/apache/datafusion-comet/pull/5415#pullrequestreview-5154502402

The original issue notes that the TPC query corpus does not contain `rlike` usage: https://github.com/apache/datafusion-comet/issues/5351

Contributor guide

Open the contributing guide

Research direction

Start by locating CometRegex and the handling of rlike, then review PR #5415 and issue #5351 for whitelist behavior and context. Identify a representative corpus and define collection, deduplication, admission, rejection grouping, and complexity measurements; done means a reproducible report covering the requested counts and distributions.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
data, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.