[VL] use libhdfs.so when exit spark will disconnect hdfs is error
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
[Environment configuration]
spark3.4.4
gluten (main and v1.3)
backend velox
hadoop:3.3.3
jdk: (open jdk)1.8.0_441 and (bisheng jdk)1.8.0_352
[how do]
run tpcds data test, execute the sql “select * from date_dim limit 10” ,it's show the result, when exit spark (ctrl + c or quit) is error.
run this cmd
spark-sql --properties-file $SPARK_HOME/conf/spark-defaults.conf_used --master local
[Expected behavior]
when used openjdk 1.8.0_441 , exit spark , log show :"hdfs disconnect failure in HdfsReadFile close:"
when used bisheng jdk, exit spark ,is coredump.
**### use openjdk logs:**
25/02/26 13:42:50 INFO SparkContext: Successfully stopped SparkContext
I20250226 13:42:50.562902 1967003 EventBase.cpp:810] EventBase(): Received terminateLoopSoon() command.
I20250226 13:42:50.563019 1966988 EventBase.cpp:794] EventBase 0xfff028008cc0 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551578 next 18446744073709551579
I20250226 13:42:50.563082 1966988 EventBase.cpp:1058] latest 18446744073709551578 next 18446744073709551579
I20250226 13:42:50.563097 1966988 EventBase.cpp:801] EventBase 0xfff028008cc0 virtual void folly::EventBase::bumpHandlingTime() (loop) startWork_ 39996343990598151
I20250226 13:42:50.563117 1966988 EventBase.cpp:794] EventBase 0xfff028008cc0 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551579 next 18446744073709551579
I20250226 13:42:50.563129 1966988 EventBase.cpp:1058] latest 18446744073709551579 next 18446744073709551579
I20250226 13:42:50.563169 1966988 EventBase.cpp:1058] latest 18446744073709551579 next 18446744073709551579
I20250226 13:42:50.563146 1966988 EventBase.cpp:655] EventBase 0xfff028008cc0 did not timeout loop time guess: 3470412 idle time: 3470369 busy time: 43 avgLoopTime: 65.9659 maxLatencyLoopTime: 65.9659 maxLatency_: 0us notificationQueueSize: 0 nothingHandledYet(): 0
I20250226 13:42:50.563189 1966988 EventBase.cpp:686] EventBase 0xfff028008cc0 loop time: 3470
I20250226 13:42:50.563200 1966988 EventBase.cpp:709] EventBase(): Done with loop.
I20250226 13:42:50.563225 1966988 EventBase.cpp:794] EventBase 0xfff028008cc0 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551579 next 18446744073709551579
I20250226 13:42:50.563235 1966988 EventBase.cpp:1058] latest 18446744073709551579 next 18446744073709551579
I20250226 13:42:50.563261 1966988 EventBase.cpp:374] EventBase(): Destroyed.
25/02/26 13:42:50 INFO ShutdownHookManager: Shutdown hook called
25/02/26 13:42:50 INFO ShutdownHookManager: Deleting directory /tmp/spark-d8abde0a-5264-4e18-91ea-1f0ba86f4ab1
25/02/26 13:42:50 INFO ShutdownHookManager: Deleting directory /tmp/spark-3bf78d2e-2fae-4dc1-bc8a-3f28a9c769e0
25/02/26 13:42:50 INFO ShutdownHookManager: Deleting directory /tmp/spark-d563ea5c-ac87-41b3-b731-46a3d4bd225d
I20250226 13:42:50.916486 1966758 HdfsFileSystem.cpp:71] Disconnecting HDFS file system
Call to AttachCurrentThread failed with error: -1
getJNIEnv: getGlobalJNIEnv failed
W20250226 13:42:50.916615 1966758 HdfsFileSystem.cpp:81] hdfs disconnect failure in HdfsReadFile close: 255
**### use bisheng jdk logs:**
25/02/26 13:46:53 INFO SparkContext: Successfully stopped SparkContext
I20250226 13:46:53.624395 1967784 EventBase.cpp:810] EventBase(): Received terminateLoopSoon() command.
I20250226 13:46:53.624492 1967774 EventBase.cpp:794] EventBase 0xffefc8001340 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551578 next 18446744073709551579
I20250226 13:46:53.624589 1967774 EventBase.cpp:1058] latest 18446744073709551578 next 18446744073709551579
I20250226 13:46:53.624603 1967774 EventBase.cpp:801] EventBase 0xffefc8001340 virtual void folly::EventBase::bumpHandlingTime() (loop) startWork_ 39996587052105286
I20250226 13:46:53.624621 1967774 EventBase.cpp:794] EventBase 0xffefc8001340 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551579 next 18446744073709551579
I20250226 13:46:53.624634 1967774 EventBase.cpp:1058] latest 18446744073709551579 next 18446744073709551579
I20250226 13:46:53.624663 1967774 EventBase.cpp:1058] latest 18446744073709551579 next 18446744073709551579
I20250226 13:46:53.624645 1967774 EventBase.cpp:655] EventBase 0xffefc8001340 did not timeout loop time guess: 2824868 idle time: 2824827 busy time: 41 avgLoopTime: -15790 maxLatencyLoopTime: -15790 maxLatency_: 0us notificationQueueSize: 0 nothingHandledYet(): 0
I20250226 13:46:53.624680 1967774 EventBase.cpp:686] EventBase 0xffefc8001340 loop time: 2824
I20250226 13:46:53.624689 1967774 EventBase.cpp:709] EventBase(): Done with loop.
I20250226 13:46:53.624713 1967774 EventBase.cpp:794] EventBase 0xffefc8001340 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551579 next 18446744073709551579
I20250226 13:46:53.624725 1967774 EventBase.cpp:1058] latest 18446744073709551579 next 18446744073709551579
I20250226 13:46:53.624748 1967774 EventBase.cpp:374] EventBase(): Destroyed.
25/02/26 13:46:53 INFO ShutdownHookManager: Shutdown hook called
25/02/26 13:46:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-d782fc63-ef7c-4565-ab1c-72c2cfe5f016
25/02/26 13:46:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-c04cad6e-534f-49e8-aee4-d0a3affca857
25/02/26 13:46:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-494937b8-f21a-401f-9595-709c1e9bf4dc
I20250226 13:46:53.972867 1967630 HdfsFileSystem.cpp:71] Disconnecting HDFS file system
Aborted
coredump:
Stack: [0x0000fff1152a0000,0x0000fff1154a0000], sp=0x0000fff11549d760, free space=2037k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libhdfs.so+0x3568] methodIdFromClass+0xc8
C [libhdfs.so+0x35d0] invokeMethodOnJclass+0x50
C [libhdfs.so+0x3bcc] invokeMethod+0xcc
C [libhdfs.so+0x6844] hdfsDisconnect+0x44
C [libvelox.so+0xa55f604] facebook::velox::filesystems::arrow::io::internal::LibHdfsShim::Disconnect(hdfs_internal*)+0x20
C [libvelox.so+0xa34459c] facebook::velox::filesystems::HdfsFileSystem::Impl::~Impl()+0x68
C [libvelox.so+0xa346018] void std::_Destroy(facebook::velox::filesystems::HdfsFileSystem::Impl*)+0x14
[Analysis]
I thought the jvm has been free when execute hdfs disconnect.
from the log ,I find “getJNIEnv: getGlobalJNIEnv failed” and “25/02/26 13:42:50 INFO ShutdownHookManager: Shutdown hook called”
from spark code:
// Add a shutdown hook to delete the temp dirs when the JVM exits
logDebug("Adding shutdown hook") // force eager creation of logger
addShutdownHook(TEMP_DIR_SHUTDOWN_PRIORITY) { () =>
### Spark version
Spark-3.4.x
### Spark configurations
spark.plugins=org.apache.gluten.GlutenPlugin
spark.driver.extraClassPath=/gluten_jars/gluten-velox-bundle-spark3.4_2.12.jar
spark.executor.extraClassPath=gluten-velox-bundle-spark3.4_2.12.jar
spark.jars=/gluten_jars/gluten-velox-bundle-spark3.4_2.12.jar
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.gluten.loadLibFromJar=false
spark.gluten.sql.debug=true
spark.gluten.sql.columnar.backend.lib=velox
spark.gluten.sql.columnar.backend.velox.memoryCapRatio=0.75
spark.gluten.sql.columnar.logicalJoinOptimizeEnable=true
spark.gluten.sql.columnar.logicalJoinOptimizationLevel=19
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Reproduce the failure with the provided spark-sql command and inspect HdfsFileSystem.cpp around the disconnect log, along with LibHdfsShim::Disconnect and HdfsReadFile close in the stack trace. Compare shutdown ordering with Spark's ShutdownHookManager and the libhdfs.so hdfsDisconnect call. Done means Spark exits without the HDFS disconnect error or coredump after the JVM shuts down.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100