[VL] GlutenParquetSchemaSuite read file timestamp as bigint schema evolusion
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
SPARK-40819: parquet file with TIMESTAMP(NANOS, true) (with nanosAsLong=true)
```
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Converted type TIMESTAMP is not allowed for requested type BIGINT
Retriable: False
Expression: !requestedType || isCompatible( requestedType, isRepeated, [](const TypePtr& type) { return type->kind() == TypeKind::TIMESTAMP; })
Context: Split [Hive: file:///Users/chengchengjin/code/spark/sql/core/src/test/resources/test-data/timestamp-nanos.parquet 0 - 784] Task Gluten_Stage_12_TID_17_VTID_8
Additional Context: Operator: TableScan[0] 0
10:56:31.680 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 12.0 (TID 17)
org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Converted type TIMESTAMP is not allowed for requested type BIGINT
Retriable: False
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start with GlutenParquetSchemaSuite and the referenced test-data/timestamp-nanos.parquet from SPARK-40819, then reproduce the TIMESTAMP(NANOS, true) read with nanosAsLong=true. Done means the file is read as the requested BIGINT schema without the Velox INVALID_STATE error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100