apache / apache/gluten

[VL] Large timestamp outside of range

Open
#7,109 5 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

Timestamps that great then 32767 years without but do not overflow will be outside of range.

``` scala
Seq(
(1723240800675l)
).toDF("time")

spark.sql("""
select from_unixtime(event_timestamp, 'HH') from test
""")

// vanilla: 56577-04-30 00:11:15
// gluten: Timestamp is outside of supported range of [-32767-01-01, 32767-12-31]
```

It may be because `std::chrono::time_point` is used in time conversion, which has range limitations.

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Scala reproduction using the large timestamp and from_unixtime, then trace the Velox timestamp-conversion path where std::chrono::time_point is used. Compare the result with vanilla Spark for the 56577-04-30 value. Done means valid timestamps beyond 32767 years no longer produce the supported-range error, with coverage for this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, scala, spark
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.