apache / apache/gluten

[VL] Limit the result of the Split function when combined with Slice

Open
#10,277 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

In our production environment, there is a poor performance case whose SQL is like `select SLICE(SPLIT(business_area_id, "\\044"), 1, 1000) from table`. As we can see, although slice only takes 1000 values, the split function needs to compute all the results. This occurs unnecessary costs when the column can be splited to more than 1000 results.

I think we can push the parameter of slice function to split. The following is a plan to demonstrate it.
original expression:
`slice(split(column1, delimiter, -1), S, N)`
transformed expression:
`slice(split(column1, delimiter, N+S-1), S, N)`

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Start by locating the Split and Slice function entry points and tracing how the production SQL expression is evaluated. Compare the original and proposed transformations for different S and N values, including the example with N=1000; done means the limited Split produces the same Slice result while avoiding unnecessary results beyond the requested range.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.