Locally built velox fails with java.io.FileNotFoundException: libboost_context.so.1.74.0 when following guide
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
Expected behaviour:
### Spark version
Spark-3.3.x
### Spark configurations
Spark 3.5
```bash
spark-sql --master 'local[5]' --conf spark.eventLog.enabled=true --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog --conf spark.sql.catalog.spark_catalog.type=hive --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.local.type=hadoop --conf spark.sql.catalog.local.warehouse=/home/holden/repos/high-performance-spark-examples/warehouse --conf spark.plugins=io.glutenproject.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=20g --jars ./gluten/package/target/gluten-velox-bundle-spark3.5_2.12-ubuntu_22.04-1.1.0-SNAPSHOT.jar, --conf spark.jars=./gluten/package/target/gluten-velox-bundle-spark3.5_2.12-ubuntu_22.04-1.1.0-SNAPSHOT.jar --conf spark.driver.extraClassPath=./gluten/package/target/gluten-package-1.1.0-SNAPSHOT.jar --conf spark.executor.extraClassPath=./gluten/package/target/gluten-package-1.1.0-SNAPSHOT.jar --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=libhigh-performance-spark-gluten-0.so --conf spark.gluten.loadLibFromJar=true --files ./native/src/libhigh-performance-spark-gluten-0.so --name
```
### System information
_No response_
### Relevant logs
```bash
park.gluten.sql.columnar.backend.velox.udfLibraryPaths=libhigh-performance-spark-gluten-0.so --conf spark.gluten.loadLibFromJar=true --files ./native/src/libhigh-performance-spark-gluten-0.so --name '' -f ''
23/11/27 18:24:45 WARN Utils: Your hostname, hda resolves to a loopback address: 127.0.1.1; using 10.69.200.180 instead (on interface enp5s0)
23/11/27 18:24:45 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
23/11/27 18:24:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/11/27 18:24:46 ERROR SparkContext: Error initializing SparkContext.
io.glutenproject.exception.GlutenException: java.io.FileNotFoundException: libboost_context.so.1.74.0
at io.glutenproject.vectorized.JniLibLoader$JniLoadTransaction.lambda$commit$0(JniLibLoader.java:253)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at io.glutenproject.vectorized.JniLibLoader$JniLoadTransaction.commit(JniLibLoader.java:256)
at io.glutenproject.utils.SharedLibraryLoaderUbuntu2204.loadLib(SharedLibraryLoaderUbuntu2204.scala:50)
at io.glutenproject.backendsapi.velox.ListenerApiImpl.loadLibFromJar(ListenerApiImpl.scala:92)
at io.glutenproject.backendsapi.velox.ListenerApiImpl.loadLibWithLinux(ListenerApiImpl.scala:101)
at io.glutenproject.backendsapi.velox.ListenerApiImpl.initialize(ListenerApiImpl.scala:135)
at io.glutenproject.backendsapi.velox.ListenerApiImpl.onDriverStart(ListenerApiImpl.scala:47)
at io.glutenproject.GlutenDriverPlugin.init(GlutenPlugin.scala:75)
at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$driverPlugins$1(PluginContainer.scala:53)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at org.apache.spark.internal.plugin.DriverPluginContainer.(PluginContainer.scala:46)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:210)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:193)
at org.apache.spark.SparkContext.(SparkContext.scala:579)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:64)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:441)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:177)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.FileNotFoundException: libboost_context.so.1.74.0
at io.glutenproject.vectorized.JniLibLoader$JniLoadTransaction.moveToWorkDir(JniLibLoader.java:298)
at io.glutenproject.vectorized.JniLibLoader$JniLoadTransaction.lambda$commit$0(JniLibLoader.java:249)
... 47 more
```
Contributor guide
Research direction
Start by reproducing the locally built Velox setup from the guide and trace the failure through JniLibLoader.java, SharedLibraryLoaderUbuntu2204.scala, and ListenerApiImpl.scala. Check how libboost_context.so.1.74.0 is located during loadLibFromJar; done means the documented command initializes Spark without the FileNotFoundException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100