apache / apache/gluten

[VL] Regexp function produces inconsistent result with Spark when using "\\s" pattern to match white space

Open
#5,105 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Reproducible test in Velox code.
```c++
diff --git a/velox/functions/sparksql/tests/RegexFunctionsTest.cpp b/velox/functions/sparksql/tests/RegexFunctionsTest.cpp
index 35bd63bfb..96cdb06b2 100644
--- a/velox/functions/sparksql/tests/RegexFunctionsTest.cpp
+++ b/velox/functions/sparksql/tests/RegexFunctionsTest.cpp
@@ -320,8 +320,8 @@ TEST_F(RegexFunctionsTest, regexpReplaceMatchSparkSqlTestSimple) {
}

TEST_F(RegexFunctionsTest, regexpReplaceWithEmptyString) {
- std::string output = "bc";
- auto result = testRegexpReplace("abc", "a", "");
+ std::string output = "1234a";
+ auto result = testRegexpReplace("1234\v", "\\s", "a");
EXPECT_EQ(result, output);
}
```

In the above test, the expected Spark's result cannot be produced by Velox.

According the discussion in the below link, RE2 community treats it an expected behavior, as they think "\v" is not a white space.

Ref.
https://github.com/google/re2j/issues/50

Contributor guide

Open the contributing guide

Research direction

Start with velox/functions/sparksql/tests/RegexFunctionsTest.cpp and the regexp replacement test shown in the issue. Trace the Velox regexp implementation and compare its handling of the "\\s" pattern against the documented Spark result and linked RE2 discussion. Done means a focused regression test passes with the expected Spark-compatible output.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, spark
Domain
backend, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.