[CH] Add support for date32 type parameters to the function get_timestamp
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
CH (ClickHouse)
### Bug description
Add support for date32 type parameters to function get_timestamp
The get_timestamp function of vanilla spark supports the first parameter being of date32 type. but CH does not.
when we query the sql like
```
SELECT count(to_date(UPD_DATE,'yyyy-MM-dd')),
count(to_date(UPD_DATE,'yyyy-MM')),
count(to_date(UPD_DATE,'yyyy'))
from TEST_CATEGORY_GROUPINGS
```
and UPD_DATE is type of date will trigger the error below :
Caused by: io.glutenproject.exception.GlutenException: Illegal type of argument #1 'time' of function parseDateTimeInJodaSyntaxOrNull, expected String, got Date32
0. /home/biao/kylin/ClickHouse/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b63ca42 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
1. /home/biao/kylin/ClickHouse/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b63ca0d in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
2. /home/biao/kylin/ClickHouse/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000021926d60 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
3. /home/biao/kylin/ClickHouse/src/Common/Exception.cpp:101: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000013f329ae in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
4. /home/biao/kylin/ClickHouse/src/Common/Exception.h:85: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b61944a in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
5. /home/biao/kylin/ClickHouse/src/Common/Exception.h:113: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type, std::type_identity::type, std::type_identity::type>, unsigned long&&, String&&, String&&, String&&, String&&) @ 0x0000000019f295da in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
6. /home/biao/kylin/ClickHouse/src/Functions/FunctionHelpers.cpp:129: DB::(anonymous namespace)::validateArgumentsImpl(DB::IFunction const&, std::vector> const&, unsigned long, std::vector> const&) @ 0x0000000019f2797e in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
7. /home/biao/kylin/ClickHouse/src/Functions/FunctionHelpers.cpp:192: DB::validateFunctionArgumentTypes(DB::IFunction const&, std::vector> const&, std::vector> const&, std::vector> const&) @ 0x0000000019f27100 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
8. /home/biao/kylin/ClickHouse/src/Functions/parseDateTime.cpp:497: DB::(anonymous namespace)::FunctionParseDateTimeImpl::getReturnTypeImpl(std::vector> const&) const @ 0x0000000011ed44c6 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
9. /home/biao/kylin/ClickHouse/src/Functions/IFunctionAdaptors.h:134: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector> const&) const @ 0x000000000b6de7b0 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
10. /home/biao/kylin/ClickHouse/src/Functions/IFunction.cpp:482: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector> const&) const @ 0x0000000019f1a4c3 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
11. /home/biao/kylin/ClickHouse/src/Functions/IFunction.cpp:433: DB::IFunctionOverloadResolver::getReturnType(std::vector> const&) const @ 0x0000000019f1a036 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
12. /home/biao/kylin/ClickHouse/src/Functions/IFunction.cpp:448: DB::IFunctionOverloadResolver::build(std::vector> const&) const @ 0x0000000019f1a641 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
13. /home/biao/kylin/ClickHouse/src/Interpreters/ActionsDAG.cpp:198: DB::ActionsDAG::addFunction(std::shared_ptr const&, std::vector>, String) @ 0x000000001a9b5326 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
14. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:1026: local_engine::SerializedPlanParser::parseFunctionWithDAG(substrait::Expression const&, String&, std::shared_ptr, bool) @ 0x0000000014353ad0 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
15. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:1728: local_engine::SerializedPlanParser::parseExpression(std::shared_ptr, substrait::Expression const&) @ 0x000000001434d41e in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
16. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:1644: local_engine::SerializedPlanParser::parseExpression(std::shared_ptr, substrait::Expression const&) @ 0x000000001434b79d in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
17. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:1288: local_engine::SerializedPlanParser::parseFunctionArgument(std::shared_ptr&, String const&, substrait::FunctionArgument const&) @ 0x000000001435f52a in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
18. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:1272: local_engine::SerializedPlanParser::parseFunctionArgument(std::shared_ptr&, std::vector>&, String const&, substrait::FunctionArgument const&) @ 0x000000001435ce99 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
19. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:1262: local_engine::SerializedPlanParser::parseFunctionArguments(std::shared_ptr&, std::vector>&, String&, substrait::Expression_ScalarFunction const&) @ 0x000000001435b7e2 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
20. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:956: local_engine::SerializedPlanParser::parseFunctionWithDAG(substrait::Expression const&, String&, std::shared_ptr, bool) @ 0x00000000143527a0 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
21. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:1323: local_engine::SerializedPlanParser::parseFunction(DB::Block const&, substrait::Expression const&, String&, std::shared_ptr, bool) @ 0x000000001434b236 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
22. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:174: local_engine::SerializedPlanParser::expressionsToActionsDAG(std::vector> const&, DB::Block const&, DB::Block const&) @ 0x0000000014349793 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
23. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/RelParser.h:73: local_engine::RelParser::expressionsToActionsDAG(std::vector> const&, DB::Block const&) @ 0x00000000143378cc in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
24. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/ProjectRelParser.cpp:61: local_engine::ProjectRelParser::parseProject(std::unique_ptr>, substrait::Rel const&, std::list>&) @ 0x000000001433213e in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
25. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/ProjectRelParser.cpp:35: local_engine::ProjectRelParser::parse(std::unique_ptr>, substrait::Rel const&, std::list>&) @ 0x0000000014331d72 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
26. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/RelParser.cpp:69: local_engine::RelParser::parseOp(substrait::Rel const&, std::list>&) @ 0x0000000014307bdb in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
27. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:579: local_engine::SerializedPlanParser::parseOp(substrait::Rel const&, std::list>&) @ 0x00000000143562d9 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
28. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:532: local_engine::SerializedPlanParser::parseOp(substrait::Rel const&, std::list>&) @ 0x0000000014355c25 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
29. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:451: local_engine::SerializedPlanParser::parse(std::unique_ptr>) @ 0x0000000014354c17 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
30. /home/biao/kylin/ClickHouse/utils/extern-local-engine/Parser/SerializedPlanParser.cpp:1830: local_engine::SerializedPlanParser::parse(String const&) @ 0x0000000014360315 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
31. /home/biao/kylin/ClickHouse/utils/extern-local-engine/local_engine_jni.cpp:255: Java_io_glutenproject_vectorized_ExpressionEvaluatorJniWrapper_nativeCreateKernelWithIterator @ 0x000000000b5e64d9 in /home/biao/kylin/ClickHouse/cmake-build-debug/utils/extern-local-engine/libchd.so
at io.glutenproject.vectorized.ExpressionEvaluatorJniWrapper.nativeCreateKernelWithIterator(Native Method)
at io.glutenproject.vectorized.CHNativeExpressionEvaluator.createKernelWithBatchIterator(CHNativeExpressionEvaluator.java:86)
at io.glutenproject.backendsapi.clickhouse.CHIteratorApi.$anonfun$genFirstStageIterator$1(CHIteratorApi.scala:128)
at io.glutenproject.utils.Arm$.withResource(Arm.scala:25)
at io.glutenproject.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37)
at io.glutenproject.backendsapi.clickhouse.CHIteratorApi.genFirstStageIterator(CHIteratorApi.scala:123)
at io.glutenproject.execution.GlutenWholeStageColumnarRDD.compute(GlutenWholeStageColumnarRDD.scala:118)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
### Spark version
Spark-3.3.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Start with src/Functions/parseDateTime.cpp and the ClickHouse function path shown in the stack trace, then trace how utils/extern-local-engine/Parser/SerializedPlanParser.cpp constructs the call. Verify the reported query with a Date32 argument and confirm that get_timestamp accepts it without the Date32 type error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, spark
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100