apache / apache/gluten

[CH] java.lang.UnsupportedOperationException: Not supported operator ShuffleQueryStage for BroadcastRelation

Open
#4,326 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

SQL: d_669_0
Driver log:
```
24/01/09 09:13:41,887 INFO [dag-scheduler-event-loop] YarnScheduler: Killing all running tasks in stage 40: Stage cancelled
24/01/09 09:13:41,887 ERROR [main] FileFormatWriter: Aborting job ef27cbe5-a6d3-425a-9799-2149acef9d23.
java.lang.UnsupportedOperationException: Not supported operator ShuffleQueryStage for BroadcastRelation
at io.glutenproject.backendsapi.clickhouse.CHSparkPlanExecApi.createBroadcastRelation(CHSparkPlanExecApi.scala:314) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:?]
at org.apache.spark.sql.execution.ColumnarBroadcastExchangeExec.$anonfun$relationFuture$2(ColumnarBroadcastExchangeExec.scala:79) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:3.3.2]
at io.glutenproject.utils.Arm$.withResource(Arm.scala:25) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:?]
at io.glutenproject.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:?]
at org.apache.spark.sql.execution.ColumnarBroadcastExchangeExec.$anonfun$relationFuture$1(ColumnarBroadcastExchangeExec.scala:69) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:3.3.2]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$1(SQLExecution.scala:191) ~[spark-sql_2.12-3.3.2.jar:3.3.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
24/01/09 09:13:41,895 INFO [dag-scheduler-event-loop] YarnScheduler: Stage 40 was cancelled
```

Contributor guide

Open the contributing guide

Research direction

Start with CHSparkPlanExecApi.scala at line 314 and trace the BroadcastRelation path from ColumnarBroadcastExchangeExec.scala. Run SQL d_669_0 with the reported Spark 3.3.2 and Gluten 1.2.0-SNAPSHOT setup, then confirm the query no longer fails with the unsupported ShuffleQueryStage exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.