apache / apache/gluten

[VL] Strip default comparator from array_sort for Velox offloading

Open
#12,094 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

When Spark's array_sort is called without an explicit comparator (the default ascending sort), Spark internally
generates a LambdaFunction that wraps the simple ascending comparison in null-handling If/IsNull logic. Velox's
rewriteArraySortCall uses a SimpleComparisonMatcher that cannot parse this null-handling wrapper, causing the
expression to fall back to Vanilla Spark.

### Gluten version

main branch

### Spark version

spark-4.0.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start at rewriteArraySortCall and inspect how SimpleComparisonMatcher handles the LambdaFunction generated for array_sort without an explicit comparator. Reproduce the default ascending sort on Spark 4.0.x, then verify that the expression is offloaded to Velox instead of falling back to Vanilla Spark.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.