microsoft / microsoft/SynapseML
UnsatisfiedLinkError OSX
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
**Describe the bug**
Trying to run tests on OSX 10.14.6, I'm getting the following error indicating the shared library is failing to load.
```
Cause: java.lang.UnsatisfiedLinkError: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives3454868880605903181/lib_lightgbm.dylib: dlopen(/private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives3454868880605903181/lib_lightgbm.dylib, 1): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
[info] Referenced from: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives3454868880605903181/lib_lightgbm.dylib
[info] Reason: image not found
```
**To Reproduce**
`sbt "testOnly com.microsoft.ml.spark.lightgbm.split2.VerifyLightGBMRanker"`
**Expected behavior**
Tests to pass
**Info (please complete the following information):**
- Latest master
- OSX 10.14.6
** Stacktrace**
```
[info] + Test Verify LightGBM Ranker on ranking dataset took 0.198s
19/08/26 13:21:24 ERROR Executor: Exception in task 3.0 in stage 16.0 (TID 49)
java.lang.UnsatisfiedLinkError: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives997331245052213810/lib_lightgbm.dylib: dlopen(/private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives997331245052213810/lib_lightgbm.dylib, 1): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives997331245052213810/lib_lightgbm.dylib
Reason: image not found
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at com.microsoft.ml.spark.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:59)
at com.microsoft.ml.spark.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:40)
at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$15.apply(TrainUtils.scala:404)
at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$15.apply(TrainUtils.scala:401)
at com.microsoft.ml.spark.core.env.StreamUtilities$.using(StreamUtilities.scala:28)
at com.microsoft.ml.spark.lightgbm.TrainUtils$.trainLightGBM(TrainUtils.scala:400)
at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$6.apply(LightGBMBase.scala:128)
at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$6.apply(LightGBMBase.scala:128)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
19/08/26 13:21:24 ERROR Executor: Exception in task 0.0 in stage 16.0 (TID 46)
```
Contributor guide
Research direction
Reproduce the failure with sbt "testOnly com.microsoft.ml.spark.lightgbm.split2.VerifyLightGBMRanker" and start at NativeLoader.loadLibraryByName and LightGBMUtils.scala:40. Trace how the LightGBM native library is loaded on macOS and inspect the related native-library packaging or build configuration. Done means the reported ranking test passes on the stated macOS environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, scala, spark
- Domain
- machine-learning, operating-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100