[CH] Map key must be a type
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
CH (ClickHouse)
### Bug description
SELECT map(1.0, '2', 3.0, '4')
[Expected behavior]
return
{1.0:"2",3.0:"4"}
[actual behavior]
throw an exception:
io.glutenproject.exception.GlutenException: Type of Map key must be a type, that can be represented by integer or String or FixedString (possibly LowCardinality) or UUID or IPv6, but Decimal(2, 1) given
### Spark version
Spark-3.3.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2668)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2604)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2603)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2603)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1178)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1178)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1178)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2856)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2798)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2787)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2238)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2259)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2278)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2303)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1021)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1020)
at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:424)
at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:451)
at org.apache.spark.sql.execution.HiveResult$.hiveResultString(HiveResult.scala:76)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.$anonfun$run$2(SparkSQLDriver.scala:69)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:69)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:286)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:984)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:191)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:214)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1072)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1081)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.glutenproject.exception.GlutenException: Type of Map key must be a type, that can be represented by integer or String or FixedString (possibly LowCardinality) or UUID or IPv6, but Decimal(2, 1) given
0. /root/code/gluten/cpp-ch/build/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000b70833b in /home/omm/libch.so
1. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x0000000005c19f03 in /home/omm/libch.so
2. /root/code/gluten/cpp-ch/build/./src/DataTypes/DataTypeMap.cpp:0: DB::DataTypeMap::assertKeyType() const @ 0x000000000efd4dca in /home/omm/libch.so
3. /root/code/gluten/cpp-ch/build/./src/DataTypes/DataTypeMap.cpp:65: DB::DataTypeMap::DataTypeMap(std::shared_ptr const&, std::shared_ptr const&) @ 0x000000000efd586a in /home/omm/libch.so
4. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1460: std::shared_ptr std::allocate_shared[abi:v15000], std::shared_ptr&, std::shared_ptr&, void>(std::allocator const&, std::shared_ptr&, std::shared_ptr&) @ 0x000000000ba3bf7e in /home/omm/libch.so
5. ./contrib/llvm-project/libcxx/include/__utility/swap.h:37: local_engine::SerializedPlanParser::parseLiteral(substrait::Expression_Literal const&) @ 0x000000000ba292f6 in /home/omm/libch.so
6. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:612: local_engine::SerializedPlanParser::parseExpression(std::shared_ptr, substrait::Expression const&) @ 0x000000000ba140fe in /home/omm/libch.so
7. ./contrib/llvm-project/libcxx/include/vector:1595: local_engine::SerializedPlanParser::parseFunctionArgument(std::shared_ptr&, std::vector>&, String const&, substrait::FunctionArgument const&) @ 0x000000000ba2741c in /home/omm/libch.so
8. ./contrib/google-protobuf/src/google/protobuf/repeated_ptr_field.h:1704: local_engine::SerializedPlanParser::parseFunctionArguments(std::shared_ptr&, std::vector>&, String&, substrait::Expression_ScalarFunction const&) @ 0x000000000ba225ad in /home/omm/libch.so
9. ./contrib/llvm-project/libcxx/include/string:1499: local_engine::SerializedPlanParser::parseFunctionWithDAG(substrait::Expression const&, String&, std::shared_ptr, bool) @ 0x000000000ba19e14 in /home/omm/libch.so
10. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:613: local_engine::SerializedPlanParser::expressionsToActionsDAG(std::vector> const&, DB::Block const&, DB::Block const&) @ 0x000000000ba0e277 in /home/omm/libch.so
11. /root/code/gluten/cpp-ch/build/./utils/extern-local-engine/Parser/ProjectRelParser.cpp:61: local_engine::ProjectRelParser::parse(std::unique_ptr>, substrait::Rel const&, std::list>&) @ 0x000000000b9f8061 in /home/omm/libch.so
12. /root/code/gluten/cpp-ch/build/./utils/extern-local-engine/Parser/RelParser.cpp:73: local_engine::RelParser::parseOp(substrait::Rel const&, std::list>&) @ 0x000000000b9d199b in /home/omm/libch.so
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: local_engine::SerializedPlanParser::parseOp(substrait::Rel const&, std::list>&) @ 0x000000000ba1d1b1 in /home/omm/libch.so
14. /root/code/gluten/cpp-ch/build/./utils/extern-local-engine/Parser/SerializedPlanParser.cpp:0: local_engine::SerializedPlanParser::parse(String const&) @ 0x000000000ba2e70d in /home/omm/libch.so
15. /root/code/gluten/cpp-ch/build/./utils/extern-local-engine/local_engine_jni.cpp:277: Java_io_glutenproject_vectorized_ExpressionEvaluatorJniWrapper_nativeCreateKernelWithIterator @ 0x0000000005f4a986 in /home/omm/libch.so
```
Contributor guide
Research direction
Start by reproducing SELECT map(1.0, '2', 3.0, '4') with Spark 3.3.x, then read SerializedPlanParser::parseLiteral in utils/extern-local-engine/Parser and DataTypeMap::assertKeyType in cpp-ch/src/DataTypes/DataTypeMap.cpp. Trace how the Decimal key reaches ClickHouse and verify that the query returns the expected map without an exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, cpp, scala
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100