ClickHouse / ClickHouse/ClickHouse

optimize_rewrite_regexp_functions strips a greedy ^.* prefix from extract patterns, silently changing which occurrence is captured (last → first)

Open
#116,922 0 comments 0 reactions 2 assignees Claimed by @vdimir View on GitHub
bug comp-query-analyzer comp-regular-function
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
17h 34m
Merged PRs (30d)
479

Description

**Describe what's wrong**

`optimize_rewrite_regexp_functions` (ON by default since 25.8) strips a leading greedy `^.*` from the pattern of `extract` (`src/Analyzer/Passes/RegexpFunctionRewritePass.cpp`, `handleExtract`). This changes which occurrence is captured whenever the rest of the pattern can match at more than one offset: the greedy `^.*` consumes as much as possible and then backtracks, so the capture group binds at the LAST offset where the remainder matches, while the stripped pattern captures at the FIRST such offset. The query silently returns a different value at pure default settings.

Only the analyzer path is affected: with `enable_analyzer = 0` both arms return the correct result.

**Does it reproduce on the most recent release?**

Yes. Verified on 26.4.1.1 and on a 26.9 development build; the `^.*` prefix stripping in `handleExtract` is present unchanged on current master. The setting has defaulted to ON since 25.8.

**How to reproduce**

```sql
SELECT
extract(h, '^.*(\\d)') AS last_digit,
extract(h, '^.*(\\d+)') AS last_digits
FROM (SELECT 'a1b2c3' AS h)
FORMAT Vertical;

-- At default settings (optimize_rewrite_regexp_functions = 1):
-- last_digit: '1'
-- last_digits: '1'
--
-- SETTINGS optimize_rewrite_regexp_functions = 0:
-- last_digit: '3'
-- last_digits: '3'
```

The pattern `^.*(\d)` is the idiomatic way to capture the last digit of a string, and with the setting OFF (as well as with re2 directly, or in any other regex engine) it returns `3`. With the rewrite the pattern becomes `(\d)` and returns the first digit instead.

The haystack has to come from a column (here a subquery alias); with a fully constant argument the expression is folded before the divergence is observable.

`EXPLAIN QUERY TREE` confirms the mechanism: with the setting ON the plan contains `extract` with the constant pattern `(\d)` instead of `^.*(\d)`.

**Expected behavior**

`extract(h, '^.*(\d)')` over `'a1b2c3'` returns `3` regardless of the optimization setting.

**Error message and/or stacktrace**

No error — the results are silently wrong.

**Additional context**

Stripping an unescaped trailing `.*$` (the other half of the same rewrite) does not have this problem, because leftmost-first matching fixes the match start before the tail is considered. The unsound half is the greedy prefix: removing `^.*` is only semantics-preserving when the remainder of the pattern can match at a single offset per input (for example, when it is itself anchored, or starts with a first-character class disjoint from what precedes it — conditions the pass does not check).

A related unsound rewrite in the same pass for `replaceRegexpAll` is reported separately: #116921.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.