apache / apache/gluten

Legacy Parser Policy doesn't exist for Velox

Open
#6,415 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

In Spark, there is legacy time parser policy(spark.sql.legacy.timeParserPolicy).
- If LEGACY, SimpleDateFormat is used for formatting and parsing dates/timestamps in a locale-sensitive manner
- If CORRECTED, classes from java.time.* packages are used for the same purpose.
- EXCEPTION, RuntimeException is thrown when we will get different results.

**Repro**:
After adding cast(timestamp -> date),
test("test to_date") {
runQueryAndCompare("select to_date('1965-01-01 10:11:12.123456','yyyy-MM-dd') as ts") { _ => }
}

This unit test fails because `spark.sql.legacy.timeParserPolicy` is configured to `exception`, which Spark adheres to. However, in Velox, the equivalent Legacy Parser Policy is absent, causing it to return null instead.

### Spark version

Spark-3.4.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.