apache / apache/gluten

[CH] Map key must be a type

Open
#4,734 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

CH (ClickHouse)

### Bug description

SELECT map(1.0, '2', 3.0, '4')
[Expected behavior]
return
{1.0:"2",3.0:"4"}
[actual behavior]
throw an exception:
io.glutenproject.exception.GlutenException: Type of Map key must be a type, that can be represented by integer or String or FixedString (possibly LowCardinality) or UUID or IPv6, but Decimal(2, 1) given

### Spark version

Spark-3.3.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2668)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2604)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2603)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2603)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1178)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1178)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1178)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2856)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2798)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2787)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2238)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2259)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2278)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2303)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1021)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1020)
at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:424)
at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:451)
at org.apache.spark.sql.execution.HiveResult$.hiveResultString(HiveResult.scala:76)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.$anonfun$run$2(SparkSQLDriver.scala:69)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:69)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:286)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:984)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:191)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:214)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1072)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1081)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.glutenproject.exception.GlutenException: Type of Map key must be a type, that can be represented by integer or String or FixedString (possibly LowCardinality) or UUID or IPv6, but Decimal(2, 1) given
0. /root/code/gluten/cpp-ch/build/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000b70833b in /home/omm/libch.so
1. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x0000000005c19f03 in /home/omm/libch.so
2. /root/code/gluten/cpp-ch/build/./src/DataTypes/DataTypeMap.cpp:0: DB::DataTypeMap::assertKeyType() const @ 0x000000000efd4dca in /home/omm/libch.so
3. /root/code/gluten/cpp-ch/build/./src/DataTypes/DataTypeMap.cpp:65: DB::DataTypeMap::DataTypeMap(std::shared_ptr const&, std::shared_ptr const&) @ 0x000000000efd586a in /home/omm/libch.so
4. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1460: std::shared_ptr std::allocate_shared[abi:v15000], std::shared_ptr&, std::shared_ptr&, void>(std::allocator const&, std::shared_ptr&, std::shared_ptr&) @ 0x000000000ba3bf7e in /home/omm/libch.so
5. ./contrib/llvm-project/libcxx/include/__utility/swap.h:37: local_engine::SerializedPlanParser::parseLiteral(substrait::Expression_Literal const&) @ 0x000000000ba292f6 in /home/omm/libch.so
6. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:612: local_engine::SerializedPlanParser::parseExpression(std::shared_ptr, substrait::Expression const&) @ 0x000000000ba140fe in /home/omm/libch.so
7. ./contrib/llvm-project/libcxx/include/vector:1595: local_engine::SerializedPlanParser::parseFunctionArgument(std::shared_ptr&, std::vector>&, String const&, substrait::FunctionArgument const&) @ 0x000000000ba2741c in /home/omm/libch.so
8. ./contrib/google-protobuf/src/google/protobuf/repeated_ptr_field.h:1704: local_engine::SerializedPlanParser::parseFunctionArguments(std::shared_ptr&, std::vector>&, String&, substrait::Expression_ScalarFunction const&) @ 0x000000000ba225ad in /home/omm/libch.so
9. ./contrib/llvm-project/libcxx/include/string:1499: local_engine::SerializedPlanParser::parseFunctionWithDAG(substrait::Expression const&, String&, std::shared_ptr, bool) @ 0x000000000ba19e14 in /home/omm/libch.so
10. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:613: local_engine::SerializedPlanParser::expressionsToActionsDAG(std::vector> const&, DB::Block const&, DB::Block const&) @ 0x000000000ba0e277 in /home/omm/libch.so
11. /root/code/gluten/cpp-ch/build/./utils/extern-local-engine/Parser/ProjectRelParser.cpp:61: local_engine::ProjectRelParser::parse(std::unique_ptr>, substrait::Rel const&, std::list>&) @ 0x000000000b9f8061 in /home/omm/libch.so
12. /root/code/gluten/cpp-ch/build/./utils/extern-local-engine/Parser/RelParser.cpp:73: local_engine::RelParser::parseOp(substrait::Rel const&, std::list>&) @ 0x000000000b9d199b in /home/omm/libch.so
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: local_engine::SerializedPlanParser::parseOp(substrait::Rel const&, std::list>&) @ 0x000000000ba1d1b1 in /home/omm/libch.so
14. /root/code/gluten/cpp-ch/build/./utils/extern-local-engine/Parser/SerializedPlanParser.cpp:0: local_engine::SerializedPlanParser::parse(String const&) @ 0x000000000ba2e70d in /home/omm/libch.so
15. /root/code/gluten/cpp-ch/build/./utils/extern-local-engine/local_engine_jni.cpp:277: Java_io_glutenproject_vectorized_ExpressionEvaluatorJniWrapper_nativeCreateKernelWithIterator @ 0x0000000005f4a986 in /home/omm/libch.so
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing SELECT map(1.0, '2', 3.0, '4') with Spark 3.3.x, then read SerializedPlanParser::parseLiteral in utils/extern-local-engine/Parser and DataTypeMap::assertKeyType in cpp-ch/src/DataTypes/DataTypeMap.cpp. Trace how the Decimal key reaches ClickHouse and verify that the query returns the expected map without an exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, cpp, scala
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.