apache / apache/gluten

[VL] Vanilla Spark occasionally failed in CI TPC-DS Q72 test

Open
#11,363 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Vanilla Spark is returning N/A (likely failing/timing out) on q72 with Spark 4.0
• Gluten is returning 100 rows, but the comparison fails because Vanilla Spark didn't complete
• This suggests a Spark 4.0 compatibility issue with q72, not necessarily a Gluten bug

link to results: https://github.com/apache/incubator-gluten/actions/runs/20706726998/job/59444584666#step:8:3461

### Gluten version

Gluten-1.5, main branch

### Spark version

spark-4.0.x

### Spark configurations

[tpc-test-ubuntu (ubuntu:22.04, spark-4.0, java-21)](https://github.com/apache/incubator-gluten/actions/runs/20706726998/job/59444584666#logs)

### System information

_No response_

### Relevant logs

```bash
Failed queries (a failed query with correct row count indicates value mismatches):

| Query ID | Passed | Row Count | Planning Time (Millis) | Query Time (Millis) | Speedup |
| | | Vanilla | Gluten | Vanilla | Gluten | Vanilla | Gluten | |
|----------|--------|---------|--------|------------|------------|-----------|-----------|---------|
| q72| false| N/A| 100| N/A| 187| N/A| 2353| N/A|

08:16:15.450 [shutdown-hook-0] WARN org.apache.spark.memory.GlobalOffHeapMemoryTarget - Memory manager not found because the code is unlikely be run in a Spark application
Error: Process completed with exit code 255.
```

Contributor guide

Open the contributing guide

Research direction

Start with the linked CI run and its q72 logs, reproducing the tpc-test-ubuntu Spark 4.0 configuration with Java 21. Compare Vanilla Spark and Gluten execution and determine whether q72 completes or times out; done means the CI comparison passes or the Spark 4.0 compatibility failure is clearly isolated and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
ci-cd, data-engineering, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.