apache / apache/gluten

[VL] [BUG] Complex types already supported in Velox are considered not supported by Gluten

Open
#11,746 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

The following SQL statement is used for the Hive write test.

```
DROP TABLE IF EXISTS hive_support;
CREATE TABLE IF NOT EXISTS hive_support (
int_col INT,
array_col ARRAY,
map_col MAP,
struct_col STRUCT
)
PARTITIONED BY (part_col INT)
STORED AS PARQUET
TBLPROPERTIES (
'parquet.compression' = 'ZSTD',
'parquet.compression.zstd.level' = '5',
'parquet.enable.dictionary' = 'false'
);
WITH number_seq AS (
SELECT CAST(id AS INT) AS id_int
FROM range(1, 50000000, 1, 100)
)
INSERT OVERWRITE TABLE hive_support
PARTITION (part_col)
SELECT
id_int AS int_col,
ARRAY(
id_int % 10,
id_int % 100,
id_int % 1000
) AS array_col,
MAP(
CONCAT('key_', CAST(id_int % 5 AS STRING)), id_int % 100,
CONCAT('key_', CAST((id_int + 1) % 5 AS STRING)), (id_int + 1) % 100
) AS map_col,
named_struct(
'field1', id_int,
'field2', CONCAT('str_', CAST(id_int AS STRING)),
'field3', id_int % 2 = 0
) AS struct_col,
id_int % 10 AS part_col
FROM number_seq
SORT BY id_int;
```

The following error is reported during the execution:
```
26/03/10 20:50:46 WARN GlutenFallbackReporter: Validation failed for plan: WriteFiles[QueryId=4], due to: Unsupported native write: Found unsupported type:ArrayType,MapType,StructType.
```

The code in `backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala` is as follows:
```
// Validate if all types are supported.
def validateDataTypes(): Option[String] = {
val unsupportedTypes = format match {
case _: ParquetFileFormat =>
fields.flatMap {
case StructField(_, _: YearMonthIntervalType, _, _) =>
Some("YearMonthIntervalType")
case StructField(_, _: StructType, _, _) =>
Some("StructType")
case _ => None
}
case _ =>
fields.flatMap {
field =>
field.dataType match {
case _: StructType => Some("StructType") // here 1
case _: ArrayType => Some("ArrayType") // here 2
case _: MapType => Some("MapType") // here 3
case _: YearMonthIntervalType => Some("YearMonthIntervalType")
case _ => None
}
}
}
if (unsupportedTypes.nonEmpty) {
Some(unsupportedTypes.mkString("Found unsupported type:", ",", ""))
} else {
None
}
}
```
After the three lines of the related type are commented out, no rollback occurs and the data is successfully written.

The Velox version downloaded by gluten in the `branch-1.3` is `https://github.com/oap-project/velox/tree/gluten-1.3.0`. In this case, `HiveDataSink::appendData` does not support `PARTITIONED BY` and complex types. However, the Velox version downloaded by gluten in the `main` branch is `https://github.com/IBM/velox/tree/dft-2026_03_10-iceberg`, where `HiveDataSink::appendData` method has been updated.

I also want to confirm whether there are any other situations that have not been considered.

### Gluten version

main branch

### Spark version

Spark-3.4.x

### Spark configurations

```
--master yarn
--driver-cores 4
--driver-memory 8g
--num-executors 12
--executor-cores 4
--executor-memory 5g
--conf spark.memory.offHeap.enabled=true
--conf spark.memory.offHeap.size=20g
--conf spark.executor.memoryOverhead=5g
--conf spark.task.cpus=1
--conf spark.executor.extraJavaOptions="-XX:+UseG1GC -XX:ActiveProcessorCount=4 -Dio.netty.tryReflectionSetAccessible=true"
--conf spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"
--conf spark.locality.wait=0
--conf spark.driver.extraClassPath="${JAR_PATH}"
--conf spark.executor.extraClassPath="${JAR_PATH}"
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
--conf spark.plugins=org.apache.gluten.GlutenPlugin
--conf spark.gluten.loadLibFromJar=false
--conf spark.gluten.sql.columnar.backend.lib=velox
--conf spark.gluten.sql.columnar.maxBatchSize=8192
--conf spark.gluten.sql.orc.charType.scan.fallback.enabled=false
--conf spark.gluten.sql.columnar.physicalJoinOptimizeEnable=true
--conf spark.gluten.sql.columnar.physicalJoinOptimizationLevel=19
--conf spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleInput=false
--conf spark.sql.adaptive.coalescePartitions.initialPartitionNum=48
--conf spark.default.parallelism=144
--conf spark.sql.shuffle.partitions=144
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer
--conf spark.sql.sources.parallelPartitionDiscovery.parallelism=60
--conf spark.network.timeout=600
--conf spark.sql.broadcastTimeout=600
--conf spark.sql.adaptive.enabled=false
--conf spark.sql.optimizer.runtime.bloomFilter.enabled=true
--conf spark.sql.hive.convertMetastoreParquet=false
--conf spark.sql.parquet.writeLegacyFormat=true
--conf spark.sql.hive.manageFilesourcePartitions=false
--conf spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict
--conf spark.hadoop.hive.exec.max.dynamic.partitions=1000
--conf spark.hadoop.hive.exec.max.dynamic.partitions.pernode=1000
--conf spark.sql.parquet.enableVectorizedWriter=false
--conf spark.sql.parquet.enableVectorizedReader=false
--conf spark.sql.parquet.enable.dictionary=false
--conf spark.hadoop.parquet.enable.dictionary=false
--conf spark.io.compression.codec=zstd
--conf spark.sql.parquet.compression.codec=zstd
```

### System information

Gluten Version: 1.7.0-SNAPSHOT
Commit: 625a47611770dc7551c5d129e4138e9bff83d748
CMake Version: 3.28.3
System: Linux-5.10.0-182.0.0.95.oe2203sp3.aarch64
Arch: aarch64
CPU Name: BIOS Model name: Kunpeng 920
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 12.5.0
C Compiler: /usr/bin/cc
C Compiler Version: 12.5.0
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib64/python3.9/site-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt

### Relevant logs

```bash
26/03/10 20:50:46 WARN GlutenFallbackReporter: Validation failed for plan: WriteFiles[QueryId=4], due to: Unsupported native write: Found unsupported type:ArrayType,MapType.
```

Contributor guide

Open the contributing guide

Research direction

Start in backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala and inspect validateDataTypes, then compare the referenced branch-1.3 and main Velox HiveDataSink::appendData implementations. Reproduce the Hive write test with the array, map, struct, and partitioned columns, and verify that supported complex-type writes are no longer rejected while unsupported cases still fall back correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.