apache / apache/gluten

Unable to create executor due to io.glutenproject.GlutenPlugin

Open
#4,492 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Hi there, i am trying to run some tpch queries based on Gluten. I built the Gluten from the main branch and also enabled the vcpkg and hdfs by setting --enable_hdfs=ON --enable_vcpkg=ON. When I enabled the gluten in the conf and submitted the app to the master standalone mode, the log shows only the executor from the master node could be created, the executors from the worker nodes will always fail, and only one of three nodes is actually working (which is master. (ERROR TaskSchedulerImpl: Lost an executor 36 (already removed): Unable to create executor due to io.glutenproject.GlutenPlugin)

I tried to trace back but failed. Below shows the log from the worker node and hopefully it helps. BTW i am actually using Spark 3.4. I have also tried the native way of submitting the job, everything works well. HDFS report also shows three datanodes alive.

### Spark version

Spark-3.3.x

### Spark configurations

$SPARK_HOME/bin/spark-submit \
--master spark://rcnfs:7077 \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.driver.extraClassPath=${GLUTEN_JAR} \
--conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.executorEnv.LIBHDFS3_CONF="/users/Wanyu/spark-env-install/hdfs-client.xml" \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g \
--executor-memory 4g \
--conf spark.executor.memoryOverhead=2g \
--conf spark.driver.maxResultSize=2g \
~/tpch/tpch_query/sf100/query9.py

### System information

Velox System Info v0.0.2
Commit: 3730305f21781391932b588c36b256b1b186f449
CMake Version: 3.22.1
System: Linux-5.15.0-86-generic
Arch: x86_64
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 11.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 11.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt

### Relevant logs

```bash
24/01/23 07:33:18 INFO Worker: Asked to launch executor app-20240123073218-0000/36 for Q9-SF100-limit
24/01/23 07:33:18 INFO SecurityManager: Changing view acls to: Wanyu
24/01/23 07:33:18 INFO SecurityManager: Changing modify acls to: Wanyu
24/01/23 07:33:18 INFO SecurityManager: Changing view acls groups to:
24/01/23 07:33:18 INFO SecurityManager: Changing modify acls groups to:
24/01/23 07:33:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Wanyu; groups with view permissions: EMPTY; users with modify permissions: Wanyu; groups with modify permissions: EMPTY
24/01/23 07:33:18 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java-8-openjdk-amd64/jre//bin/java" "-cp" "/users/Wanyu/gluten/package/target/gluten-velox-bundle-spark3.4_2.12-ubuntu_22.04-1.2.0-SNAPSHOT.jar:/users/Wanyu/spark/conf/:/users/Wanyu/spark/jars/*:/users/Wanyu/hadoop/etc/hadoop/:/users/Wanyu/hadoop/share/hadoop/common/lib/*:/users/Wanyu/hadoop/share/hadoop/common/*:/users/Wanyu/hadoop/share/hadoop/hdfs/:/users/Wanyu/hadoop/share/hadoop/hdfs/lib/*:/users/Wanyu/hadoop/share/hadoop/hdfs/*:/users/Wanyu/hadoop/share/hadoop/mapreduce/*:/users/Wanyu/hadoop/share/hadoop/yarn/:/users/Wanyu/hadoop/share/hadoop/yarn/lib/*:/users/Wanyu/hadoop/share/hadoop/yarn/*" "-Xmx4096M" "-Dspark.driver.port=42977" "-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "-Djdk.reflect.useDirectMethodHandle=false" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@pc728.emulab.net:42977" "--executor-id" "36" "--hostname" "155.98.36.31" "--cores" "4" "--app-id" "app-20240123073218-0000" "--worker-url" "spark://Worker@155.98.36.31:7078" "--resourceProfileId" "0"
24/01/23 07:33:21 INFO Worker: Executor app-20240123073218-0000/36 finished with state EXITED message Command exited with code 1 exitStatus 1
24/01/23 07:33:21 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 36
24/01/23 07:33:21 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20240123073218-0000, execId=36)
```

Contributor guide

Open the contributing guide

Research direction

Start with the spark-submit configuration and the worker's ExecutorRunner launch log, comparing the worker and master environments and classpaths. Reproduce the standalone submission and inspect why GlutenPlugin initialization exits on worker nodes; done means executors start successfully across all three nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.