[VL] Limit the result of the Split function when combined with Slice
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
In our production environment, there is a poor performance case whose SQL is like `select SLICE(SPLIT(business_area_id, "\\044"), 1, 1000) from table`. As we can see, although slice only takes 1000 values, the split function needs to compute all the results. This occurs unnecessary costs when the column can be splited to more than 1000 results.
I think we can push the parameter of slice function to split. The following is a plan to demonstrate it.
original expression:
`slice(split(column1, delimiter, -1), S, N)`
transformed expression:
`slice(split(column1, delimiter, N+S-1), S, N)`
### Gluten version
None
Contributor guide
Research direction
Start by locating the Split and Slice function entry points and tracing how the production SQL expression is evaluated. Compare the original and proposed transformations for different S and N values, including the example with N=1000; done means the limited Split produces the same Slice result while avoiding unnecessary results beyond the requested range.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100