[VL] Spark 4.1: Add TimeType (TIME data type) support (SPARK-51162)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
## Backend
VL (Velox)
**Gluten version**: main branch
## Description
Spark 4.1 introduced the TIME data type (SPARK-51162 SPIP, 101 sub-tasks). `DataTypeTestUtils.atomicTypes` now includes `TimeType`, causing all tests that iterate over `atomicTypes` to fail when encountering the unsupported type.
Feature toggle: `spark.sql.timeType.enabled` (default true in tests).
Spark 4.1 only.
**Parent issue**: #11910 (`[VL] Spark 4.x: Tracking new feature support`)
### Impact
| Suite | Failures | How affected |
|-------|:--------:|-------------|
| GlutenRandomDataGeneratorSuite | 232 | atomicTypes enumeration |
| GlutenOrderingSuite | 2 | TimeType(0) / TimeType(6) |
| GlutenHiveResultSuite | 1 | New test (SPARK-51517) |
| GlutenCastWithAnsiOnSuite (partial) | ~6 | TimeType cast tests |
Note: GlutenCastWithAnsiOnSuite also has 4 ANSI Cast failures — those belong to #10134, not this issue.
Additionally, some SQL query test files may be blocked by TimeType (cast.sql, literals.sql, nonansi/literals.sql).
### References
- Apache Spark JIRA: [SPARK-51162](https://issues.apache.org/jira/browse/SPARK-51162)
Contributor guide
Research direction
Start by running the affected GlutenRandomDataGeneratorSuite, GlutenOrderingSuite, GlutenHiveResultSuite, and GlutenCastWithAnsiOnSuite against Spark 4.1, then inspect failures involving TimeType. Check cast.sql, literals.sql, and nonansi/literals.sql for blocked queries. Done means the listed TimeType failures are resolved, while the unrelated ANSI Cast failures from #10134 remain separate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- backend, databases, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100