apache / apache/gluten

[VL] Found unsupported data type in OrcReadFormat

Open
#10,441 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description
```
25/08/14 08:02:29 WARN GlutenFallbackReporter: Validation failed for plan: Scan orc spark_catalog.tpcds_bin_partitioned_varchar_orc_1000.date_dim[QueryId=1], due to: Found unsupported data type in OrcReadFormat: Some(varchar(16))(force fallback), Some(varchar(9))(force fallback), Some(varchar(6))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback)..
25/08/14 08:02:29 WARN GlutenFallbackReporter: Validation failed for plan: ColumnarToRow[QueryId=1], due to: Found unsupported data type in OrcReadFormat: Some(varchar(16))(force fallback), Some(varchar(9))(force fallback), Some(varchar(6))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback)..
25/08/14 08:02:29 WARN GlutenFallbackReporter: Validation failed for plan: Scan orc spark_catalog.tpcds_bin_partitioned_varchar_orc_1000.date_dim[QueryId=1], due to: Found unsupported data type in OrcReadFormat: Some(varchar(16))(force fallback), Some(varchar(9))(force fallback), Some(varchar(6))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback)..
25/08/14 08:02:29 WARN GlutenFallbackReporter: Validation failed for plan: ColumnarToRow[QueryId=1], due to: Found unsupported data type in OrcReadFormat: Some(varchar(16))(force fallback), Some(varchar(9))(force fallback), Some(varchar(6))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback), Some(varchar(1))(force fallback)..
```

### Gluten version

Gluten-1.3

### Spark version

Spark-3.4.x

### Spark configurations

${SPARK_HOME}/bin/spark-sql \
--master yarn \
--driver-memory 20g \
--driver-cores 1 \
--executor-memory 16g \
--executor-cores 8 \
--num-executors 19 \
--database tpcds_bin_partitioned_varchar_orc_1000 \
--conf spark.sql.shuffle.partitions=320 \
--conf spark.default.parallelism=320 \
--conf spark.sql.adaptive.enabled=false \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=10G \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.driver.extraClassPath=/home/velox-test/incubator-gluten/package/target/gluten-velox-bundle-spark3.4_2.12-debian_11_aarch_64-1.3.0.jar \
--conf spark.executor.extraClassPath=/home/velox-test/incubator-gluten/package/target/gluten-velox-bundle-spark3.4_2.12-debian_11_aarch_64-1.3.0.jar \
--conf spark.executorEnv.LIBHDFS3_CONF="/home/velox-test/hadoop-3.3.1/etc/hadoop/hdfs-client.xml" \
--conf spark.driver.extraJavaOptions="
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:ConcGCThreads=10
-XX:ParallelGCThreads=20
-Xlog:gc*=info:file=${LOG_DIR}/driver_gc.log:time,pid,tags
" \
--conf spark.sql.warehouse.dir=hdfs:///user/hive/warehouse \
-f ../tpcds/spark-sql/all.sql \
2>&1 | tee ${LOG_DIR}/tpcds.log

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Reproduce the report with the Spark SQL command and ../tpcds/spark-sql/all.sql using the listed Gluten-1.3 configuration. Trace the validation path for OrcReadFormat and the reported varchar types. Done means the ORC scan no longer reports these varchar columns as unsupported and forces fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark, sql
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.