to_date result incorrect when the number of digit is smaller than 4
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
When using to_date for string-to-date, the result shows difference compared to Vanilla Spark.
Below is the reproduce step:
create table to_date as select '1609304400' as t1, '1609304054' as t2 union select '1609304400' as t1, '0' as t2;
select to_date(cast(t2 as date)),t2 from to_date;
### Spark version
Spark-3.2.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Start by running the reported CREATE TABLE and SELECT reproduction with Spark 3.2.x and compare the results. Trace the to_date string-to-date handling in the Velox backend; done means the outputs for both provided t2 values match vanilla Spark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark, sql
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100