[VL] Regexp function produces inconsistent result with Spark when using "\\s" pattern to match white space
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
Reproducible test in Velox code.
```c++
diff --git a/velox/functions/sparksql/tests/RegexFunctionsTest.cpp b/velox/functions/sparksql/tests/RegexFunctionsTest.cpp
index 35bd63bfb..96cdb06b2 100644
--- a/velox/functions/sparksql/tests/RegexFunctionsTest.cpp
+++ b/velox/functions/sparksql/tests/RegexFunctionsTest.cpp
@@ -320,8 +320,8 @@ TEST_F(RegexFunctionsTest, regexpReplaceMatchSparkSqlTestSimple) {
}
TEST_F(RegexFunctionsTest, regexpReplaceWithEmptyString) {
- std::string output = "bc";
- auto result = testRegexpReplace("abc", "a", "");
+ std::string output = "1234a";
+ auto result = testRegexpReplace("1234\v", "\\s", "a");
EXPECT_EQ(result, output);
}
```
In the above test, the expected Spark's result cannot be produced by Velox.
According the discussion in the below link, RE2 community treats it an expected behavior, as they think "\v" is not a white space.
Ref.
https://github.com/google/re2j/issues/50
Contributor guide
Research direction
Start with velox/functions/sparksql/tests/RegexFunctionsTest.cpp and the regexp replacement test shown in the issue. Trace the Velox regexp implementation and compare its handling of the "\\s" pattern against the documented Spark result and linked RE2 discussion. Done means a focused regression test passes with the expected Spark-compatible output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, spark
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100