apache / apache/gluten

[VL] InsertIntoHadoopFsRelationCommand is used for parquet writing

Open
#4,656 3 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

I tested the GlutenVelox 1.1 release jar and also built the jar myself with `--enable_hdfs=ON`.
However when running the below query or spark-shell scala code, the `InsertIntoHadoopFsRelationCommand` is falling back to CPU:
```
In spark-shell:
val df=spark.read.parquet("xxx")
df.write.mode("overwrite").format("parquet").save("file:///tmp/yyy")
```
Or
```
In spark-sql:
insert overwrite local directory 'file:///tmp/yyy/' STORED AS parquet select col from db.tbl;
```

My expectation is the parquet write happens on Velox without `VeloxColumnarToRowExec`.

### Spark version

Spark-3.3.1

### Spark configurations

Single node Spark Standalone Cluster with a Spark Master and a Spark worker.

### System information

There is no HDFS cluster. Only a local file system

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.