[VL] Result mismatch in date format week year
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
'Y' means week-based-year in spark (SimpleDateFormat). But velox parse 'Y' as year.
``` scala
Seq(
( 883584000), // 1998-01-01 00:00:00
(1608912000), // 2020-12-26 00:00:00
(1608998400), // 2020-12-27 00:00:00
(1640361600), // 2021-12-25 00:00:00
(1640448000), // 2021-12-26 00:00:00
(1640966400), // 2022-01-01 00:00:00
(1672416000), // 2022-12-31 00:00:00
(1672502400), // 2023-01-01 00:00:00
(1703865600), // 2023-12-30 00:00:00
(1703952000), // 2023-12-31 00:00:00
).toDF("date")
spark.sql(s"""
select
from_unixtime(date, 'Y') as week_year,
date
from tmp
""")
/*
vanilla: gluten:
+---------+----------+ +---------+----------+
|week_year| date| |week_year| date|
+---------+----------+ +---------+----------+
| 1998| 883584000| | 1998| 883584000|
| 2020|1608912000| | 2020|1608912000|
| 2021|1608998400| | 2020|1608998400|
| 2021|1640361600| | 2021|1640361600|
| 2022|1640448000| | 2021|1640448000|
| 2022|1640966400| | 2022|1640966400|
| 2022|1672416000| | 2022|1672416000|
| 2023|1672502400| | 2023|1672502400|
| 2023|1703865600| | 2023|1703865600|
| 2024|1703952000| | 2023|1703952000|
+---------+----------+ +---------+----------+
*/
```
I'm trying tofix this mismatch, and there are two issues that need to be resolved:
1. JodaDateTimeFormatter interprets Y as the 'year of era', which is diff from SimpleDateFormat in java.
2. Velox uses ISO standard to calc week date (e.g. https://github.com/facebookincubator/velox/pull/10713). But SimpleDateFormat use [GregorianCalendar](https://github.com/openjdk/jdk8/blob/6a383433a9f4661a96a90b2a4c7b5b9a85720031/jdk/src/share/classes/java/util/GregorianCalendar.java#L2077).
The main difference is that SimpleDateFormat will define the firstDayOfWeek and the minimalDaysInFirstWeek based on locale. Tt is Sunday and 1 day by default. ISO8601 defines it as Monday and 4 days.
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.