apache / apache/pinot

SIMPLE_DATE_FORMAT have a different behavior for `X` letter compared to Java

Open
#9,822 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Java's `SimpleDateFormat` (https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html) has a `X` letter to represent the time zone in the format of `-08:00`. The behavior of Pinot's `SIMPLE_DATE_FORMAT` in `DateTimeFieldSpec` is actually different from Java.

```
input = 2022-11-16 15:22:37.123+00:00
SIMPLE_DATE_FORMAT|yyyy-MM-dd HH:mm:ss.SSSX <- this is the config based on Java doc but it's not working
SIMPLE_DATE_FORMAT|yyyy-MM-dd HH:mm:ss.SSSZ <- this is the working config but it's not supposed to work based on Java definition.

```

Here is the code to reproduce:

```
public static void main(String[] args)
throws ParseException {
String input = "2022-11-16 15:22:37.123+00:00";

// This is working and this is the expected behavior based on the Java documentation.
String pattern = "yyyy-MM-dd HH:mm:ss.SSSX";
SimpleDateFormat simpleDateFormat = new SimpleDateFormat(pattern);
Date date = simpleDateFormat.parse(input);
System.out.println(date.toInstant().toEpochMilli());

// This is not working. But, it is supposed to work if we follow the convention for SimpleDateFormat
try {
String format = "SIMPLE_DATE_FORMAT|yyyy-MM-dd HH:mm:ss.SSSX";
DateTimeFormatSpec dateTimeFormatSpec = new DateTimeFormatSpec(format);
long millis = dateTimeFormatSpec.getDateTimeFormatter().parseMillis(input);
System.out.println(millis);
} catch (Exception e) {
// not working
System.out.println("failed");
}

// This is working and this is the expected behavior based on the documentation from JAVA.
try {
String pattern2 = "yyyy-MM-dd HH:mm:ss.SSSZ";
simpleDateFormat = new SimpleDateFormat(pattern2);
date = simpleDateFormat.parse(input);
System.out.println(date.toInstant().toEpochMilli());
} catch (Exception e) {
// not working
System.out.println("failed");
}

// This working but Java's SimpleDateFormat's definition for `Z` doesn't include "-00:00" so it supposed to fail.
String format2 = "SIMPLE_DATE_FORMAT|yyyy-MM-dd HH:mm:ss.SSSZ";
DateTimeFormatSpec dateTimeFormatSpec = new DateTimeFormatSpec(format2);
long millis2 = dateTimeFormatSpec.getDateTimeFormatter().parseMillis(input);
System.out.println(millis2);
}

```

Contributor guide

Open the contributing guide

Research direction

Start with DateTimeFormatSpec and reproduce the reported input using SIMPLE_DATE_FORMAT with the X and Z patterns. Compare the parser behavior with Java's SimpleDateFormat documentation and verify that the supported pattern letters handle the documented timezone forms consistently; done means the reproduction behaves as expected for both patterns.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.