[VL] Vanilla Spark occasionally failed in CI TPC-DS Q72 test
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
Vanilla Spark is returning N/A (likely failing/timing out) on q72 with Spark 4.0
• Gluten is returning 100 rows, but the comparison fails because Vanilla Spark didn't complete
• This suggests a Spark 4.0 compatibility issue with q72, not necessarily a Gluten bug
link to results: https://github.com/apache/incubator-gluten/actions/runs/20706726998/job/59444584666#step:8:3461
### Gluten version
Gluten-1.5, main branch
### Spark version
spark-4.0.x
### Spark configurations
[tpc-test-ubuntu (ubuntu:22.04, spark-4.0, java-21)](https://github.com/apache/incubator-gluten/actions/runs/20706726998/job/59444584666#logs)
### System information
_No response_
### Relevant logs
```bash
Failed queries (a failed query with correct row count indicates value mismatches):
| Query ID | Passed | Row Count | Planning Time (Millis) | Query Time (Millis) | Speedup |
| | | Vanilla | Gluten | Vanilla | Gluten | Vanilla | Gluten | |
|----------|--------|---------|--------|------------|------------|-----------|-----------|---------|
| q72| false| N/A| 100| N/A| 187| N/A| 2353| N/A|
08:16:15.450 [shutdown-hook-0] WARN org.apache.spark.memory.GlobalOffHeapMemoryTarget - Memory manager not found because the code is unlikely be run in a Spark application
Error: Process completed with exit code 255.
```
Contributor guide
Research direction
Start with the linked CI run and its q72 logs, reproducing the tpc-test-ubuntu Spark 4.0 configuration with Java 21. Compare Vanilla Spark and Gluten execution and determine whether q72 completes or times out; done means the CI comparison passes or the Spark 4.0 compatibility failure is clearly isolated and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- ci-cd, data-engineering, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100