Unable to create executor due to io.glutenproject.GlutenPlugin
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
Hi there, i am trying to run some tpch queries based on Gluten. I built the Gluten from the main branch and also enabled the vcpkg and hdfs by setting --enable_hdfs=ON --enable_vcpkg=ON. When I enabled the gluten in the conf and submitted the app to the master standalone mode, the log shows only the executor from the master node could be created, the executors from the worker nodes will always fail, and only one of three nodes is actually working (which is master. (ERROR TaskSchedulerImpl: Lost an executor 36 (already removed): Unable to create executor due to io.glutenproject.GlutenPlugin)
I tried to trace back but failed. Below shows the log from the worker node and hopefully it helps. BTW i am actually using Spark 3.4. I have also tried the native way of submitting the job, everything works well. HDFS report also shows three datanodes alive.
### Spark version
Spark-3.3.x
### Spark configurations
$SPARK_HOME/bin/spark-submit \
--master spark://rcnfs:7077 \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.driver.extraClassPath=${GLUTEN_JAR} \
--conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.executorEnv.LIBHDFS3_CONF="/users/Wanyu/spark-env-install/hdfs-client.xml" \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g \
--executor-memory 4g \
--conf spark.executor.memoryOverhead=2g \
--conf spark.driver.maxResultSize=2g \
~/tpch/tpch_query/sf100/query9.py
### System information
Velox System Info v0.0.2
Commit: 3730305f21781391932b588c36b256b1b186f449
CMake Version: 3.22.1
System: Linux-5.15.0-86-generic
Arch: x86_64
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 11.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 11.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt
### Relevant logs
```bash
24/01/23 07:33:18 INFO Worker: Asked to launch executor app-20240123073218-0000/36 for Q9-SF100-limit
24/01/23 07:33:18 INFO SecurityManager: Changing view acls to: Wanyu
24/01/23 07:33:18 INFO SecurityManager: Changing modify acls to: Wanyu
24/01/23 07:33:18 INFO SecurityManager: Changing view acls groups to:
24/01/23 07:33:18 INFO SecurityManager: Changing modify acls groups to:
24/01/23 07:33:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Wanyu; groups with view permissions: EMPTY; users with modify permissions: Wanyu; groups with modify permissions: EMPTY
24/01/23 07:33:18 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java-8-openjdk-amd64/jre//bin/java" "-cp" "/users/Wanyu/gluten/package/target/gluten-velox-bundle-spark3.4_2.12-ubuntu_22.04-1.2.0-SNAPSHOT.jar:/users/Wanyu/spark/conf/:/users/Wanyu/spark/jars/*:/users/Wanyu/hadoop/etc/hadoop/:/users/Wanyu/hadoop/share/hadoop/common/lib/*:/users/Wanyu/hadoop/share/hadoop/common/*:/users/Wanyu/hadoop/share/hadoop/hdfs/:/users/Wanyu/hadoop/share/hadoop/hdfs/lib/*:/users/Wanyu/hadoop/share/hadoop/hdfs/*:/users/Wanyu/hadoop/share/hadoop/mapreduce/*:/users/Wanyu/hadoop/share/hadoop/yarn/:/users/Wanyu/hadoop/share/hadoop/yarn/lib/*:/users/Wanyu/hadoop/share/hadoop/yarn/*" "-Xmx4096M" "-Dspark.driver.port=42977" "-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "-Djdk.reflect.useDirectMethodHandle=false" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@pc728.emulab.net:42977" "--executor-id" "36" "--hostname" "155.98.36.31" "--cores" "4" "--app-id" "app-20240123073218-0000" "--worker-url" "spark://Worker@155.98.36.31:7078" "--resourceProfileId" "0"
24/01/23 07:33:21 INFO Worker: Executor app-20240123073218-0000/36 finished with state EXITED message Command exited with code 1 exitStatus 1
24/01/23 07:33:21 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 36
24/01/23 07:33:21 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20240123073218-0000, execId=36)
```
Contributor guide
Research direction
Start with the spark-submit configuration and the worker's ExecutorRunner launch log, comparing the worker and master environments and classpaths. Reproduce the standalone submission and inspect why GlutenPlugin initialization exits on worker nodes; done means executors start successfully across all three nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100