[CH] java.lang.UnsupportedOperationException: Not supported operator ShuffleQueryStage for BroadcastRelation
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
SQL: d_669_0
Driver log:
```
24/01/09 09:13:41,887 INFO [dag-scheduler-event-loop] YarnScheduler: Killing all running tasks in stage 40: Stage cancelled
24/01/09 09:13:41,887 ERROR [main] FileFormatWriter: Aborting job ef27cbe5-a6d3-425a-9799-2149acef9d23.
java.lang.UnsupportedOperationException: Not supported operator ShuffleQueryStage for BroadcastRelation
at io.glutenproject.backendsapi.clickhouse.CHSparkPlanExecApi.createBroadcastRelation(CHSparkPlanExecApi.scala:314) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:?]
at org.apache.spark.sql.execution.ColumnarBroadcastExchangeExec.$anonfun$relationFuture$2(ColumnarBroadcastExchangeExec.scala:79) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:3.3.2]
at io.glutenproject.utils.Arm$.withResource(Arm.scala:25) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:?]
at io.glutenproject.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:?]
at org.apache.spark.sql.execution.ColumnarBroadcastExchangeExec.$anonfun$relationFuture$1(ColumnarBroadcastExchangeExec.scala:69) ~[gluten-1.2.0-SNAPSHOT-spark-3.3-jar-with-dependencies.jar:3.3.2]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$1(SQLExecution.scala:191) ~[spark-sql_2.12-3.3.2.jar:3.3.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
24/01/09 09:13:41,895 INFO [dag-scheduler-event-loop] YarnScheduler: Stage 40 was cancelled
```
Contributor guide
Research direction
Start with CHSparkPlanExecApi.scala at line 314 and trace the BroadcastRelation path from ColumnarBroadcastExchangeExec.scala. Run SQL d_669_0 with the reported Spark 3.3.2 and Gluten 1.2.0-SNAPSHOT setup, then confirm the query no longer fails with the unsupported ShuffleQueryStage exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100