apache / apache/gluten

[VL] Spark 4.1: Velox split function returns incorrect results with limit parameter (SPARK-49968)

Open
#11,913 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

## Backend
VL (Velox)

## Bug description

`split('hello', '', 1)` returns `["h"]` in Velox but should return `["hello"]`. This is a Velox C++ re2-based split implementation issue when the `limit` parameter is specified. Spark 4.1 added the `limit` parameter test via SPARK-49968.

Spark 4.1 only.

Parent issue: #11550

### Impact

| Suite / Item | Status | spark40 | spark41 |
|---|---|:---:|:---:|
| GlutenCollationRegexpExpressionsSuite | Entire suite TODO (1 failure) | 🟢 | 🔴 |
| GlutenRegexpExpressionsSuite | `.exclude("SPLIT")` | 🟢 | 🔴 |
| VeloxStringFunctionsSuite | `testWithSpecifiedSparkVersion` skips 4.x | 🟢 | 🔴 |

### References

- Apache Spark JIRA: [SPARK-49968](https://issues.apache.org/jira/browse/SPARK-49968)

Contributor guide

Open the contributing guide

Research direction

Start by running the Spark 4.1 cases in GlutenCollationRegexpExpressionsSuite, GlutenRegexpExpressionsSuite, and VeloxStringFunctionsSuite, focusing on split with an empty pattern and limit 1. Trace the Velox C++ re2-based split implementation. Done means split('hello', '', 1) returns ["hello"] and the affected Spark 4.1 tests no longer fail or skip this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.