apache / apache/gluten

[VL] PySpark failed to run sql query Iceberg data.

Open
#8,278 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

I use PySpark to execute a SQL query on Iceberg data stored on HDFS, but the following exception occurs, but the same SQL can be run successfully using Java. My cluster environment has HDFS configuration information.

Error message:

**py4j.protocol.Py4JJavaError: An error occurred while calling o156.showString.
: java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 12) ( executor 3): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Unable to connect to HDFS: nameservice, got error: InvalidParameter: Cannot parse URI: hdfs://nameservice, missing port or invalid HA configuration Caused by: HdfsConfigNotFound: Config key: dfs.ha.namenodes.nameservice not found.
Retriable: False
Expression: hdfsClient_ != nullptr
Context: Split [Hive: hdfs://nameservice/spark/tpch_iceberg.db/supplier_ice/data/00000-116-410ef5e1-e2df-44fd-b67a-4a9410655fa1-00001.parquet 4 - 513211] Task Gluten_Stage_3_TID_12_VTID_1
Additional Context: Operator: TableScan[0] 0
Function: Impl
File: Gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/hdfs/HdfsFileSystem.cpp
Line: 37**

Is there any standard guidance document for using Gluten in pyspark?

### Spark version

Spark-3.4.x

### Spark configurations

_No response_

### System information

os: centos7
spark: 3.4.1

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the failure in Gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/hdfs/HdfsFileSystem.cpp at line 37, then compare the PySpark and Java execution environments and their HDFS HA configuration. Check the missing dfs.ha.namenodes.nameservice setting and URI parsing error; done means the reported PySpark SQL query reads the Iceberg data on HDFS successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, python
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.