datafusion-contrib / datafusion-contrib/StreamFusion
Native string predicates: support LIKE ESCAPE and SIMILAR TO
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Rechecked on 2026-09-20 against canonical main 89da01cdad7342c4d825e36eaf4e8dc73bc7af9f, released Apache Flink 2.2.1, JDK 17, default compatibility settings. Each expression was run against independent stock-Flink and StreamFusion-enabled streaming environments with bounded runtime DataStream input (parallelism 1), so source values cannot be folded away. Results and resolved result types agree through Flink fallback; this is a native support gap, not a demonstrated wrong-result bug.
Explicit ESCAPE causes LIKE/NOT LIKE to fall back; SIMILAR TO is also unsupported. Adapted SQL from LikeFunctionITCase and ScalarFunctionsTest plus minimal runtime probes confirms successful host execution.
Runtime row: s STRING='a_b', p STRING='a!_b', e STRING='!'.
SELECT s LIKE 'a!_b' ESCAPE '!' FROM src; -- TRUE
SELECT s LIKE p ESCAPE e FROM src; -- TRUE
SELECT s NOT LIKE 'a!_b' ESCAPE '!' FROM src; -- FALSE
All three record Calc: unsupported Calc expression. Both literal and column-dependent escape forms are affected.
For runtime s='This is a test String.':
SELECT s SIMILAR TO 'This (is)? a (test)+ Strin_*' FROM src; -- TRUE
SELECT s NOT SIMILAR TO '_*' FROM src; -- FALSE
These record Calc: unsupported function/operator: SIMILAR TO.
Verify Flink's SQL pattern grammar separately from Java regex: escaped percent/underscore, empty escapes where accepted, Unicode, NULLs, malformed escape sequences and exception timing. Preserve NOT and WHERE behavior. A specific fallback diagnostic would also make the unsupported LIKE signature easier to identify.
Suggested priority: P2, common STRING filtering.
Acceptance: add runtime-source SQL parity tests that assert native Calc execution as well as values and resolved types; include NULLs and boundary/error cases; retain safe fallback for unverified forms and update docs/operators/calc-filter.md with the admitted scope.
Latest-main validation: rebuilt the debug native library from this commit and ran 562 targeted runtime SQL cases covering these support gaps and adjacent controls. This issue group contains 51 successful comparisons that still record explicit fallback. Host-rejected/both-error cases are excluded from that count. This validates released Flink 2.2.1; it does not claim execution of the Flink 1.18 profile. No implementation is included.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the adapted LikeFunctionITCase and ScalarFunctionsTest cases, then inspect the native Calc runtime behavior for the listed LIKE and SIMILAR TO forms. Add runtime-source parity coverage for values, resolved types, NULLs, and boundary errors while preserving fallback for unverified forms. Update docs/operators/calc-filter.md with the admitted scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, rust, sql
- Domain
- data-engineering, documentation, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100