[VL] to_timestamp truncates microseconds part yyyy-MM-dd HH:mm:ss.SSSSSS
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
This UT can repro the issue
```scala
test("to_timestamp") {
spark.sql("create table abc (ts_str string) using parquet")
spark.sql("insert into abc values ('2025-02-25 03:07:22.123456')")
val df = spark.sql("select * from abc").
withColumn("timestamp", to_timestamp(col("ts_str"), "yyyy-MM-dd HH:mm:ss.SSSSSS"))
df.collect().foreach(println)
}
```
Expected result (Spark):
`[2025-02-25 03:07:22.123456,2025-02-25 03:07:22.123456]`
Actual result:
`[2025-02-25 03:07:22.123456,2025-02-25 03:07:22.123]`
Seems it shows till milliseconds only
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start with the Scala reproducer for to_timestamp on the Velox backend and trace timestamp parsing for the yyyy-MM-dd HH:mm:ss.SSSSSS pattern. Compare the parsed result with Spark's expected microsecond-preserving value and verify that the reproducer prints 2025-02-25 03:07:22.123456 rather than truncating to milliseconds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100