[VL] Spark 4.1: Velox split function returns incorrect results with limit parameter (SPARK-49968)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
## Backend
VL (Velox)
## Bug description
`split('hello', '', 1)` returns `["h"]` in Velox but should return `["hello"]`. This is a Velox C++ re2-based split implementation issue when the `limit` parameter is specified. Spark 4.1 added the `limit` parameter test via SPARK-49968.
Spark 4.1 only.
Parent issue: #11550
### Impact
| Suite / Item | Status | spark40 | spark41 |
|---|---|:---:|:---:|
| GlutenCollationRegexpExpressionsSuite | Entire suite TODO (1 failure) | 🟢 | 🔴 |
| GlutenRegexpExpressionsSuite | `.exclude("SPLIT")` | 🟢 | 🔴 |
| VeloxStringFunctionsSuite | `testWithSpecifiedSparkVersion` skips 4.x | 🟢 | 🔴 |
### References
- Apache Spark JIRA: [SPARK-49968](https://issues.apache.org/jira/browse/SPARK-49968)
Contributor guide
Research direction
Start by running the Spark 4.1 cases in GlutenCollationRegexpExpressionsSuite, GlutenRegexpExpressionsSuite, and VeloxStringFunctionsSuite, focusing on split with an empty pattern and limit 1. Trace the Velox C++ re2-based split implementation. Done means split('hello', '', 1) returns ["hello"] and the affected Spark 4.1 tests no longer fail or skip this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100