apache / apache/datafusion

support `regexp_match` as predicate

Open
#15,872 8 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

From https://github.com/openobserve/openobserve/discussions/6584
regexp_match does not seem to work with length or space matches. see below.

### To Reproduce

```
$ mkdir ~/.cargo/tmp; TMPDIR=~/.cargo/tmp cargo install datafusion-cli
$ datafusion-cli
DataFusion CLI v47.0.0
> select * from 'test-datafusion.csv';
+---+-------+
| a | b |
+---+-------+
| 1 | one |
| 2 | two |
| 3 | three |
+---+-------+
3 row(s) fetched.
Elapsed 0.006 seconds.
> select * from 'test-datafusion.csv' where regexp_match(a, '(.){4,}');
Error during planning: Cannot create filter with non-boolean predicate 'regexp_match(test-datafusion.csv.a, Utf8("(.){4,}"))' returning List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })
> select * from 'test-datafusion.csv';
+---+--------------+
| a | b |
+---+--------------+
| 1 | one |
| 2 | two |
| 3 | three |
| 3 | four -O test |
+---+--------------+
4 row(s) fetched.
Elapsed 0.010 seconds.

> select * from 'test-datafusion.csv' where regexp_match(b, '(.){4,}');
Error during planning: Cannot create filter with non-boolean predicate 'regexp_match(test-datafusion.csv.b, Utf8("(.){4,}"))' returning List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })
> select * from 'test-datafusion.csv' where regexp_match(b, '\s-O\s');
Error during planning: Cannot create filter with non-boolean predicate 'regexp_match(test-datafusion.csv.b, Utf8("\s-O\s"))' returning List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })

```

### Expected behavior

both should return one result with no error.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with datafusion-cli and the regexp_match predicates shown in the report, including the length and whitespace patterns. Trace how the predicate is planned and verify that both queries return one result without a non-boolean predicate error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.