apache / apache/gluten

[VL] GlutenParquetSchemaSuite read file timestamp as bigint schema evolusion

Open
#11,220 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

SPARK-40819: parquet file with TIMESTAMP(NANOS, true) (with nanosAsLong=true)
```
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Converted type TIMESTAMP is not allowed for requested type BIGINT
Retriable: False
Expression: !requestedType || isCompatible( requestedType, isRepeated, [](const TypePtr& type) { return type->kind() == TypeKind::TIMESTAMP; })
Context: Split [Hive: file:///Users/chengchengjin/code/spark/sql/core/src/test/resources/test-data/timestamp-nanos.parquet 0 - 784] Task Gluten_Stage_12_TID_17_VTID_8
Additional Context: Operator: TableScan[0] 0
10:56:31.680 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 12.0 (TID 17)
org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Converted type TIMESTAMP is not allowed for requested type BIGINT
Retriable: False
```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with GlutenParquetSchemaSuite and the referenced test-data/timestamp-nanos.parquet from SPARK-40819, then reproduce the TIMESTAMP(NANOS, true) read with nanosAsLong=true. Done means the file is read as the requested BIGINT schema without the Velox INVALID_STATE error.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.