datafusion-contrib / datafusion-contrib/StreamFusion
Native regex scalars: support REGEXP and REGEXP_REPLACE/COUNT/INSTR/SUBSTR
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Rechecked on 2026-09-20 against canonical main 89da01cdad7342c4d825e36eaf4e8dc73bc7af9f, released Apache Flink 2.2.1, JDK 17, default compatibility settings. Each expression was run against independent stock-Flink and StreamFusion-enabled streaming environments with bounded runtime DataStream input (parallelism 1), so source values cannot be folded away. Results and resolved result types agree through Flink fallback; this is a native support gap, not a demonstrated wrong-result bug.
The following scalar-result regular-expression functions still report Calc: unsupported function/operator: <name>. Evidence comes from adapted RegexpFunctionsITCase / ScalarFunctionsTest SQL and minimal runtime probes.
Runtime columns: s STRING='abc123abc', p STRING='abc', r STRING='X'. Execute each projection separately:
| SQL projection | Flink result |
|---|---|
| REGEXP(s,p) | TRUE |
| REGEXP_REPLACE(s,p,r) | 'X123X' |
| REGEXP_COUNT(s,p) | 2 |
| REGEXP_INSTR(s,p) | 1 |
| REGEXP_SUBSTR(s,p) | 'abc' |
Example: SELECT REGEXP_REPLACE(s,p,r) FROM src. Each returns the same result via fallback, with no native substitution.
Match released Flink's Java-regex semantics and supported overloads, including captures/replacement references, zero-width matches, Unicode, NULLs, invalid patterns and exception timing. A Rust regex engine is not automatically equivalent. Add projection and filter consumers and batch-boundary tests.
Array-returning REGEXP_EXTRACT_ALL should be scheduled after these scalar STRING/INT/BOOLEAN forms.
Suggested priority: P2, after basic numeric and string predicate support.
Acceptance: add runtime-source SQL parity tests that assert native Calc execution as well as values and resolved types; include NULLs and boundary/error cases; retain safe fallback for unverified forms and update docs/operators/calc-filter.md with the admitted scope.
Latest-main validation: rebuilt the debug native library from this commit and ran 562 targeted runtime SQL cases covering these support gaps and adjacent controls. This issue group contains 39 successful comparisons that still record explicit fallback. Host-rejected/both-error cases are excluded from that count. This validates released Flink 2.2.1; it does not claim execution of the Flink 1.18 profile. No implementation is included.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the adapted RegexpFunctionsITCase and ScalarFunctionsTest SQL, then review the native Calc support and docs/operators/calc-filter.md. Run the runtime-source parity cases for REGEXP and the scalar REGEXP_* functions. Done means native Calc execution matches Flink values and resolved types, including NULL and boundary/error cases, while unverified forms retain safe fallback and the admitted scope is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, rust, sql
- Domain
- documentation, stream-processing, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100