apache / apache/gluten

[VL] to_timestamp truncates microseconds part yyyy-MM-dd HH:mm:ss.SSSSSS

Open
#10,314 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

This UT can repro the issue
```scala
test("to_timestamp") {
spark.sql("create table abc (ts_str string) using parquet")
spark.sql("insert into abc values ('2025-02-25 03:07:22.123456')")
val df = spark.sql("select * from abc").
withColumn("timestamp", to_timestamp(col("ts_str"), "yyyy-MM-dd HH:mm:ss.SSSSSS"))
df.collect().foreach(println)
}
```

Expected result (Spark):
`[2025-02-25 03:07:22.123456,2025-02-25 03:07:22.123456]`

Actual result:
`[2025-02-25 03:07:22.123456,2025-02-25 03:07:22.123]`

Seems it shows till milliseconds only

### Gluten version

main branch

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with the Scala reproducer for to_timestamp on the Velox backend and trace timestamp parsing for the yyyy-MM-dd HH:mm:ss.SSSSSS pattern. Compare the parsed result with Spark's expected microsecond-preserving value and verify that the reproducer prints 2025-02-25 03:07:22.123456 rather than truncating to milliseconds.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.