[VL] Can not load native library using the --jars
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
when using `--conf spark.gluten.loadLibFromJar=true` to load native libraries from the jar, you need to set the parameters
` --conf spark.driver.extraClassPath=${GLUTEN_JAR}:${JNI_JAR} ` and `--conf spark.executor.extraClassPath=${GLUTEN_JAR}:${JNI_JAR} ` to add the ${JNI_JAR} to the extraClassPath. So, each node should have the path ${JIN_JAR}.
if using `--jars ${JNI_PATH}` it can cause error in native library loading;
the code throw an exception is:
https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/java/org/apache/gluten/vectorized/JniLibLoader.java#L214
the reason for not found the native library may be that, in https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/java/org/apache/gluten/vectorized/JniLibLoader.java#L212, the class Loader is the ` sun.misc.Launcher$AppClassLoader` but using `--jars ` the loader is the loader in spark `MutableURLClassLoader`
Contributor guide
Research direction
Start with gluten-core/src/main/java/org/apache/gluten/vectorized/JniLibLoader.java around lines 212-214 and compare the class loader used with --jars against the extraClassPath setup described in the issue. Reproduce native library loading with --jars; done means the library loads without the reported exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100