apache / apache/gluten

Timestamp type scan result is not correct

Open
#6,831 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Found by UT "Timestamp type" in `VeloxOrcDataTypeValidationSuite`.
```
2024-08-05T07:04:37.9907383Z - Timestamp type *** FAILED ***
2024-08-05T07:04:37.9907935Z Results do not match for query:
2024-08-05T07:04:37.9911208Z Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
2024-08-05T07:04:37.9915475Z Timezone Env:
2024-08-05T07:04:37.9915736Z
2024-08-05T07:04:37.9915962Z == Parsed Logical Plan ==
2024-08-05T07:04:37.9916335Z 'GlobalLimit 100
2024-08-05T07:04:37.9916617Z +- 'LocalLimit 100
2024-08-05T07:04:37.9916924Z +- 'Project ['timestamp]
2024-08-05T07:04:37.9917330Z +- 'UnresolvedRelation [type1], [], false
2024-08-05T07:04:37.9917684Z
2024-08-05T07:04:37.9917896Z == Analyzed Logical Plan ==
2024-08-05T07:04:37.9918191Z timestamp: timestamp
2024-08-05T07:04:37.9918455Z GlobalLimit 100
2024-08-05T07:04:37.9918729Z +- LocalLimit 100
2024-08-05T07:04:37.9919034Z +- Project [timestamp#22489]
2024-08-05T07:04:37.9919382Z +- SubqueryAlias type1
2024-08-05T07:04:37.9920450Z +- View (`type1`, [byte#22478,short#22479,int#22480,long#22481L,float#22482,double#22483,decimal#22484,string#22485,binary#22486,bool#22487,date#22488,timestamp#22489,array#22490,struct#22491,map#22492])
2024-08-05T07:04:37.9922782Z +- RelationV2[byte#22478, short#22479, int#22480, long#22481L, float#22482, double#22483, decimal#22484, string#22485, binary#22486, bool#22487, date#22488, timestamp#22489, array#22490, struct#22491, map#22492] orc file:/__w/incubator-gluten/incubator-gluten/backends-velox/target/scala-2.12/test-classes/data-type-validation-data/type1_orc
2024-08-05T07:04:37.9924240Z
2024-08-05T07:04:37.9924461Z == Optimized Logical Plan ==
2024-08-05T07:04:37.9924751Z GlobalLimit 100
2024-08-05T07:04:37.9925016Z +- LocalLimit 100
2024-08-05T07:04:37.9925902Z +- RelationV2[timestamp#22489] orc file:/__w/incubator-gluten/incubator-gluten/backends-velox/target/scala-2.12/test-classes/data-type-validation-data/type1_orc
2024-08-05T07:04:37.9927209Z
2024-08-05T07:04:37.9927563Z == Physical Plan ==
2024-08-05T07:04:37.9928020Z CollectLimit 100
2024-08-05T07:04:37.9928544Z +- VeloxColumnarToRowExec
2024-08-05T07:04:37.9932170Z +- ^(1285) BatchScanExecTransformer[timestamp#22489] OrcScan DataFilters: [], Format: orc, Location: InMemoryFileIndex(1 paths)[file:/__w/incubator-gluten/incubator-gluten/backends-velox/target/scal..., PartitionFilters: [], PushedAggregation: [], PushedFilters: [], PushedGroupBy: [], ReadSchema: struct, PushedFilters: [], PushedAggregation: [], PushedGroupBy: [] RuntimeFilters: []
2024-08-05T07:04:37.9935679Z
2024-08-05T07:04:37.9936024Z == Results ==
2024-08-05T07:04:37.9936392Z
2024-08-05T07:04:37.9936728Z == Results ==
2024-08-05T07:04:37.9937353Z !== Correct Answer - 1 == == Gluten Answer - 1 ==
2024-08-05T07:04:37.9937997Z struct<> struct<>
2024-08-05T07:04:37.9938883Z ![2013-04-08 13:10:15.0] [2013-04-08 14:10:15.0] (GlutenQueryTest.scala:325)

```

### Spark version

None

### Spark configurations

_No response_

### System information

Velox System Info v0.0.2
Commit: ac0a3ac569f21a1e245a2494c6e1a9edd78767ba
CMake Version: 3.28.3
System: Linux-5.4.0-156-generic
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 9.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 9.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib/python3.8/dist-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt

\nThe results will be copied to your clipboard if xclip is installed.

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.