apache / apache/gluten

[VL] Result mismatch in date format week year

Open
#7,069 5 comments 1 reaction 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

'Y' means week-based-year in spark (SimpleDateFormat). But velox parse 'Y' as year.

``` scala
Seq(
( 883584000), // 1998-01-01 00:00:00
(1608912000), // 2020-12-26 00:00:00
(1608998400), // 2020-12-27 00:00:00
(1640361600), // 2021-12-25 00:00:00
(1640448000), // 2021-12-26 00:00:00
(1640966400), // 2022-01-01 00:00:00
(1672416000), // 2022-12-31 00:00:00
(1672502400), // 2023-01-01 00:00:00
(1703865600), // 2023-12-30 00:00:00
(1703952000), // 2023-12-31 00:00:00
).toDF("date")

spark.sql(s"""
select
from_unixtime(date, 'Y') as week_year,
date
from tmp
""")

/*
vanilla: gluten:
+---------+----------+ +---------+----------+
|week_year| date| |week_year| date|
+---------+----------+ +---------+----------+
| 1998| 883584000| | 1998| 883584000|
| 2020|1608912000| | 2020|1608912000|
| 2021|1608998400| | 2020|1608998400|
| 2021|1640361600| | 2021|1640361600|
| 2022|1640448000| | 2021|1640448000|
| 2022|1640966400| | 2022|1640966400|
| 2022|1672416000| | 2022|1672416000|
| 2023|1672502400| | 2023|1672502400|
| 2023|1703865600| | 2023|1703865600|
| 2024|1703952000| | 2023|1703952000|
+---------+----------+ +---------+----------+
*/
```

I'm trying tofix this mismatch, and there are two issues that need to be resolved:

1. JodaDateTimeFormatter interprets Y as the 'year of era', which is diff from SimpleDateFormat in java.
2. Velox uses ISO standard to calc week date (e.g. https://github.com/facebookincubator/velox/pull/10713). But SimpleDateFormat use [GregorianCalendar](https://github.com/openjdk/jdk8/blob/6a383433a9f4661a96a90b2a4c7b5b9a85720031/jdk/src/share/classes/java/util/GregorianCalendar.java#L2077).

The main difference is that SimpleDateFormat will define the firstDayOfWeek and the minimalDaysInFirstWeek based on locale. Tt is Sunday and 1 day by default. ISO8601 defines it as Monday and 4 days.

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.