apache / apache/gluten

[VL] Read OrcFile error when schema in Orc file and the table file don't consist

Open
#5,638 10 comments 0 reactions 1 assignee Claimed by @kecookier View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

when the schema in Orc file is different from the table schema(especially the field name), the gluten read error. as follows:
schema in ORC file:
image
the user table is:
CREATE TABLE parquet_test
(
treatment TINYINT,
numerator DOUBLE,
denominator TINYINT,
numerator_pre BIGINT,
denominator_pre TINYINT,
Y DOUBLE,
X1 INT,
X2 INT,
X3 INT,
X3_string STRING,
X7_needcut BIGINT,
X8_needcut BIGINT,
weight DOUBLE,
distance DOUBLE
)

then execute the query:
val df = spark.sql( "SELECT count(Y) from udf_table")
a incorrect value returns,
image

![](https://file.tapd.woa.com/compress/compress_img/700/tapd_70077997_base64_1715042626_348.png?src=/tfl/captures/2024-05/tapd_70077997_base64_1715042626_348.png)

but use query: val df = spark.sql( "SELECT count(_col5) from udf_table")
return ok;
image

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.