apache / apache/gluten

[VL] Cannot allocate memory in static TLS block

Open
#9,036 3 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Hi, I'm trying to execute Velox Spark jobs on arm64 instances (aws graviton), but running into the error `cannot allocate memory in static TLS block`.

### Spark version

Spark-3.2.x

### Spark configurations

_No response_

### System information

Instance: r6gd.16xlarge

### Relevant logs

```text
Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/spark-local-dir/gluten-xxxx/jni/.../linux/aarch64/libvelox.so: /tmp/spark-local-dir/gluten-xxx/jni/.../gluten-xxxx/linux/aarch64/libgluten.so: cannot allocate memory in static TLS block

at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)

at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2450)

at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2506)

at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2705)

at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2635)

at java.base/java.lang.Runtime.load0(Runtime.java:768)

at java.base/java.lang.System.load(System.java:1854)

at org.apache.gluten.jni.JniLibLoader.loadFromPath0(JniLibLoader.java:96)

at org.apache.gluten.jni.JniLibLoader.loadWithLink(JniLibLoader.java:219)

at org.apache.gluten.jni.JniLibLoader.load(JniLibLoader.java:169)

at org.apache.gluten.backendsapi.velox.VeloxListenerApi.initialize(VeloxListenerApi.scala:157)

at org.apache.gluten.backendsapi.velox.VeloxListenerApi.onDriverStart(VeloxListenerApi.scala:93)

at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart(SubstraitBackend.scala:27)

at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart$(SubstraitBackend.scala:26)

at org.apache.gluten.backendsapi.velox.VeloxBackend.onDriverStart(VeloxBackend.scala:53)

at org.apache.gluten.GlutenDriverPlugin.init(GlutenPlugin.scala:71)

at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$driverPlugins$1(PluginContainer.scala:53)

at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)

at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)

at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)

at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)

at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)

at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)

at org.apache.spark.internal.plugin.DriverPluginContainer.(PluginContainer.scala:46)

at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:210)

at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:193)

at org.apache.spark.SparkContext.(SparkContext.scala:576)

at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2772)

at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:950)

at scala.Option.getOrElse(Option.scala:189)

at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:944)

at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:54)

at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:333)

at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:160)

at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1005)

at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)

at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)

at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)

at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1093)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1102)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

```

Contributor guide

Open the contributing guide

Research direction

Start with the native-library loading paths in JniLibLoader.java and VeloxListenerApi.scala, then reproduce the failure using the reported Spark 3.2.x setup on an arm64 AWS instance. Trace loading of libgluten.so and libvelox.so and confirm that Velox Spark jobs start successfully without the static TLS allocation error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, scala
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.