[VL] PySpark failed to run sql query Iceberg data.
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
I use PySpark to execute a SQL query on Iceberg data stored on HDFS, but the following exception occurs, but the same SQL can be run successfully using Java. My cluster environment has HDFS configuration information.
Error message:
**py4j.protocol.Py4JJavaError: An error occurred while calling o156.showString.
: java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 12) ( executor 3): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Unable to connect to HDFS: nameservice, got error: InvalidParameter: Cannot parse URI: hdfs://nameservice, missing port or invalid HA configuration Caused by: HdfsConfigNotFound: Config key: dfs.ha.namenodes.nameservice not found.
Retriable: False
Expression: hdfsClient_ != nullptr
Context: Split [Hive: hdfs://nameservice/spark/tpch_iceberg.db/supplier_ice/data/00000-116-410ef5e1-e2df-44fd-b67a-4a9410655fa1-00001.parquet 4 - 513211] Task Gluten_Stage_3_TID_12_VTID_1
Additional Context: Operator: TableScan[0] 0
Function: Impl
File: Gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/hdfs/HdfsFileSystem.cpp
Line: 37**
Is there any standard guidance document for using Gluten in pyspark?
### Spark version
Spark-3.4.x
### Spark configurations
_No response_
### System information
os: centos7
spark: 3.4.1
### Relevant logs
_No response_
Contributor guide
Research direction
Start with the failure in Gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/hdfs/HdfsFileSystem.cpp at line 37, then compare the PySpark and Java execution environments and their HDFS HA configuration. Check the missing dfs.ha.namenodes.nameservice setting and URI parsing error; done means the reported PySpark SQL query reads the Iceberg data on HDFS successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, python
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100