apache / apache/gluten

[VL] TPC-H Q15 outputs unstable query results

Open
#3,561 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

[Expected behavior] and [actual behavior].
When I use velox as the backend execution engine to test TPC-H, the output result is unstable. Sometimes it prints one result but sometimes the result is empty like below:
![image](https://github.com/oap-project/gluten/assets/89506884/f1834694-a34e-4a66-a4a2-dfa09ef86493)

### Spark version

Spark-3.2.x

### Spark configurations

cat tpch_parquet.scala | ${SPARK_HOME}/bin/spark-shell \
--master yarn --deploy-mode client \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.gluten.sql.columnar.backend.lib=velox \
--conf spark.driver.extraClassPath=${GLUTEN_JAR} \
--conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g\
--conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--num-executors 8 \
--executor-cores 4 \
--driver-memory 10g \
--executor-memory 5g \
--conf spark.executor.memoryOverhead=5g \
--conf spark.driver.maxResultSize=2g \
--conf spark.sql.files.maxPartitionBytes=2g \
--conf spark.sql.shuffle.partitions=192

### System information

Velox System Info v0.0.2
Commit: 0fd70fff3d79c643058f826a7ab83cdf9c141098
CMake Version: 3.27.2
System: Linux-5.15.0-spr.bkc.pc.16.4.24.x86_64
Arch: x86_64
C++ Compiler: /opt/rh/gcc-toolset-11/root/usr/bin/c++
C++ Compiler Version: 11.2.1
C Compiler: /opt/rh/gcc-toolset-11/root/usr/bin/cc
C Compiler Version: 11.2.1
CMake Prefix Path: /usr/local;/usr;/;/home/weiqiang/.local/lib/python3.8/site-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Run tpch_parquet.scala through spark-shell with the listed Spark and Velox configuration, repeating TPC-H Q15 to confirm whether the output changes or becomes empty. Compare the repeated results and relevant logs; done means Q15 produces the same correct, non-empty output consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.