[VL] Cannot allocate memory in static TLS block
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
Hi, I'm trying to execute Velox Spark jobs on arm64 instances (aws graviton), but running into the error `cannot allocate memory in static TLS block`.
### Spark version
Spark-3.2.x
### Spark configurations
_No response_
### System information
Instance: r6gd.16xlarge
### Relevant logs
```text
Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/spark-local-dir/gluten-xxxx/jni/.../linux/aarch64/libvelox.so: /tmp/spark-local-dir/gluten-xxx/jni/.../gluten-xxxx/linux/aarch64/libgluten.so: cannot allocate memory in static TLS block
at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2450)
at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2506)
at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2705)
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2635)
at java.base/java.lang.Runtime.load0(Runtime.java:768)
at java.base/java.lang.System.load(System.java:1854)
at org.apache.gluten.jni.JniLibLoader.loadFromPath0(JniLibLoader.java:96)
at org.apache.gluten.jni.JniLibLoader.loadWithLink(JniLibLoader.java:219)
at org.apache.gluten.jni.JniLibLoader.load(JniLibLoader.java:169)
at org.apache.gluten.backendsapi.velox.VeloxListenerApi.initialize(VeloxListenerApi.scala:157)
at org.apache.gluten.backendsapi.velox.VeloxListenerApi.onDriverStart(VeloxListenerApi.scala:93)
at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart(SubstraitBackend.scala:27)
at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart$(SubstraitBackend.scala:26)
at org.apache.gluten.backendsapi.velox.VeloxBackend.onDriverStart(VeloxBackend.scala:53)
at org.apache.gluten.GlutenDriverPlugin.init(GlutenPlugin.scala:71)
at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$driverPlugins$1(PluginContainer.scala:53)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at org.apache.spark.internal.plugin.DriverPluginContainer.(PluginContainer.scala:46)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:210)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:193)
at org.apache.spark.SparkContext.(SparkContext.scala:576)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2772)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:950)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:944)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:54)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:333)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:160)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1005)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1093)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1102)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
Contributor guide
Research direction
Start with the native-library loading paths in JniLibLoader.java and VeloxListenerApi.scala, then reproduce the failure using the reported Spark 3.2.x setup on an arm64 AWS instance. Trace loading of libgluten.so and libvelox.so and confirm that Velox Spark jobs start successfully without the static TLS allocation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, scala
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100