[VL]1941-12-25 00:00:00 is in a gap between
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
The query contains a project expression
`cast(datediff('2026-01-19', from_unixtime(unix_timestamp(a.birthdate,'yyyyMMdd'), 'yyyy-MM-dd'))/365 as int)`
It causes the error show below.
```
Caused by: org.apache.gluten.exception.GlutenException: Exception: VeloxUserError
Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: 1941-12-25 00:00:00 is in a gap between
1941-12-25 00:00:00 HKWT and
1941-12-25 00:30:00 JST which are both equivalent to
1941-12-24 15:30:00 UTC
Retriable: False
Context: unix_timestamp(n0_13, yyyyMMdd:VARCHAR)
Additional Context: Top-level Expression: try_cast((try_cast((divide(try_cast((datediff(2026-01-19:DATE, try_cast((from_unixtime(unix_timestamp(n0_13, yyyyMMdd:VARCHAR), yyyy-MM-dd:VARCHAR)) as DATE))) as DOUBLE), 365:DOUBLE)) as INTEGER)) as VARCHAR)
Function: toGMT
File: /home/hadoop/gluten/ep/build-velox/build/velox_ep/velox/type/Timestamp.cpp
Line: 64
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_14VeloxUserErrorEPKcEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox9Timestamp5toGMTERKNS0_2tz8TimeZoneE.cold
# 4 _ZZN8facebook5velox4bits10forEachBitIZNS0_4exec7EvalCtx22applyToSelectedNoThrowIZNKS3_21SimpleFunctionAdapterINS0_4core9UDFHolderINS0_9functions8sparksql36UnixTimestampParseWithFormatFunctionINS3_10VectorExecEEESC_lNS0_15ConstantCheckerIJNS0_7VarcharESF_EEEJSF_SF_EEEE7iterateIJNS3_16FlatVectorReaderISF_EENS3_20ConstantVectorReaderISF_EEEEEvRNSI_12ApplyContextEDpRT_EUlT_E3_ZNS4_22applyToSelectedNoThrowISU_EEvRKNS0_17SelectivityVectorEST_EUlST_E_EEvSY_ST_T0_EUlST_E_EEvPKmiibST_ENKUliE0_clEi
# 5 _ZN8facebook5velox4bits10forEachBitIZNS0_4exec7EvalCtx22applyToSelectedNoThrowIZNKS3_21SimpleFunctionAdapterINS0_4core9UDFHolderINS0_9functions8sparksql36UnixTimestampParseWithFormatFunctionINS3_10VectorExecEEESC_lNS0_15ConstantCheckerIJNS0_7VarcharESF_EEEJSF_SF_EEEE7iterateIJNS3_16FlatVectorReaderISF_EENS3_20ConstantVectorReaderISF_EEEEEvRNSI_12ApplyContextEDpRT_EUlT_E3_ZNS4_22applyToSelectedNoThrowISU_EEvRKNS0_17SelectivityVectorEST_EUlST_E_EEvSY_ST_T0_EUlST_E_EEvPKmiibST_
# 6 _ZNK8facebook5velox4exec21SimpleFunctionAdapterINS0_4core9UDFHolderINS0_9functions8sparksql36UnixTimestampParseWithFormatFunctionINS1_10VectorExecEEES8_lNS0_15ConstantCheckerIJNS0_7VarcharESB_EEEJSB_SB_EEEE5applyERKNS0_17SelectivityVectorERSt6vectorISt10shared_ptrINS0_10BaseVectorEESaISL_EERKSJ_IKNS0_4TypeEERNS1_7EvalCtxERSL_
# 7 _ZN8facebook5velox4exec4Expr13applyFunctionERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 8 _ZN8facebook5velox4exec4Expr11evalAllImplERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 9 _ZN8facebook5velox4exec4Expr4evalERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEEPKNS1_7ExprSetE
# 10 _ZN8facebook5velox4exec4Expr11evalAllImplERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 11 _ZN8facebook5velox4exec4Expr4evalERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEEPKNS1_7ExprSetE
# 12 _ZN8facebook5velox4exec8CastExpr15evalSpecialFormERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 13 _ZN8facebook5velox4exec4Expr24evalSpecialFormWithStatsERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 14 _ZN8facebook5velox4exec4Expr11evalAllImplERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 15 _ZN8facebook5velox4exec4Expr4evalERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEEPKNS1_7ExprSetE
# 16 _ZN8facebook5velox4exec4Expr11evalAllImplERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 17 _ZN8facebook5velox4exec4Expr4evalERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEEPKNS1_7ExprSetE
# 18 _ZN8facebook5velox4exec8CastExpr15evalSpecialFormERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 19 _ZN8facebook5velox4exec4Expr24evalSpecialFormWithStatsERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 20 _ZN8facebook5velox4exec4Expr11evalAllImplERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 21 _ZN8facebook5velox4exec4Expr4evalERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEEPKNS1_7ExprSetE
# 22 _ZN8facebook5velox4exec4Expr11evalAllImplERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 23 _ZN8facebook5velox4exec4Expr4evalERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEEPKNS1_7ExprSetE
# 24 _ZN8facebook5velox4exec8CastExpr15evalSpecialFormERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 25 _ZN8facebook5velox4exec4Expr24evalSpecialFormWithStatsERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 26 _ZN8facebook5velox4exec4Expr11evalAllImplERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 27 _ZN8facebook5velox4exec4Expr4evalERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEEPKNS1_7ExprSetE
# 28 _ZN8facebook5velox4exec8CastExpr15evalSpecialFormERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 29 _ZN8facebook5velox4exec4Expr24evalSpecialFormWithStatsERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 30 _ZN8facebook5velox4exec4Expr11evalAllImplERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 31 _ZN8facebook5velox4exec4Expr13evalWithNullsERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEE
# 32 _ZN8facebook5velox4exec4Expr4evalERKNS0_17SelectivityVectorERNS1_7EvalCtxERSt10shared_ptrINS0_10BaseVectorEEPKNS1_7ExprSetE
# 33 _ZN8facebook5velox4exec7ExprSet4evalEiibRKNS0_17SelectivityVectorERNS1_7EvalCtxERSt6vectorISt10shared_ptrINS0_10BaseVectorEESaISB_EE
# 34 _ZN8facebook5velox4exec13FilterProject7projectERKNS0_17SelectivityVectorERNS1_7EvalCtxE
# 35 _ZN8facebook5velox4exec13FilterProject9getOutputEv
# 36 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv
# 37 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 38 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 39 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 40 _ZN6gluten24WholeStageResultIterator4nextEv
# 41 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 42 0x00007fba3517dad3
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
... 33 more
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
... 21 more
Driver stacktrace:
```
### Gluten version
1.3.0
### Spark version
3.5.2
### Spark configurations
### System information
### Relevant logs
Contributor guide
Research direction
Reproduce the query and inspect Velox Timestamp.cpp around line 64, following the unix_timestamp path shown in the error. Check the related Gluten execution flow near ColumnarBatchOutIterator.java:57 and add coverage for this timezone-gap case; done means the reported expression has defined behavior without the current INVALID_ARGUMENT failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, scala, spark
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100