[VL] Large timestamp outside of range
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
Timestamps that great then 32767 years without but do not overflow will be outside of range.
``` scala
Seq(
(1723240800675l)
).toDF("time")
spark.sql("""
select from_unixtime(event_timestamp, 'HH') from test
""")
// vanilla: 56577-04-30 00:11:15
// gluten: Timestamp is outside of supported range of [-32767-01-01, 32767-12-31]
```
It may be because `std::chrono::time_point` is used in time conversion, which has range limitations.
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Start with the Scala reproduction using the large timestamp and from_unixtime, then trace the Velox timestamp-conversion path where std::chrono::time_point is used. Compare the result with vanilla Spark for the 56577-04-30 value. Done means valid timestamps beyond 32767 years no longer produce the supported-range error, with coverage for this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, scala, spark
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100