apache / apache/gluten

[VL] Spark 4.1: Add TimeType (TIME data type) support (SPARK-51162)

Open
#11,919 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

## Backend
VL (Velox)

**Gluten version**: main branch

## Description

Spark 4.1 introduced the TIME data type (SPARK-51162 SPIP, 101 sub-tasks). `DataTypeTestUtils.atomicTypes` now includes `TimeType`, causing all tests that iterate over `atomicTypes` to fail when encountering the unsupported type.

Feature toggle: `spark.sql.timeType.enabled` (default true in tests).

Spark 4.1 only.

**Parent issue**: #11910 (`[VL] Spark 4.x: Tracking new feature support`)

### Impact

| Suite | Failures | How affected |
|-------|:--------:|-------------|
| GlutenRandomDataGeneratorSuite | 232 | atomicTypes enumeration |
| GlutenOrderingSuite | 2 | TimeType(0) / TimeType(6) |
| GlutenHiveResultSuite | 1 | New test (SPARK-51517) |
| GlutenCastWithAnsiOnSuite (partial) | ~6 | TimeType cast tests |

Note: GlutenCastWithAnsiOnSuite also has 4 ANSI Cast failures — those belong to #10134, not this issue.

Additionally, some SQL query test files may be blocked by TimeType (cast.sql, literals.sql, nonansi/literals.sql).

### References

- Apache Spark JIRA: [SPARK-51162](https://issues.apache.org/jira/browse/SPARK-51162)

Contributor guide

Open the contributing guide

Research direction

Start by running the affected GlutenRandomDataGeneratorSuite, GlutenOrderingSuite, GlutenHiveResultSuite, and GlutenCastWithAnsiOnSuite against Spark 4.1, then inspect failures involving TimeType. Check cast.sql, literals.sql, and nonansi/literals.sql for blocked queries. Done means the listed TimeType failures are resolved, while the unrelated ANSI Cast failures from #10134 remain separate.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
backend, databases, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.