microsoft / microsoft/SynapseML

UnsatisfiedLinkError OSX

Open
#668 4 comments 0 reactions 0 assignees View on GitHub
area/lightgbm
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

**Describe the bug**
Trying to run tests on OSX 10.14.6, I'm getting the following error indicating the shared library is failing to load.

```
Cause: java.lang.UnsatisfiedLinkError: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives3454868880605903181/lib_lightgbm.dylib: dlopen(/private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives3454868880605903181/lib_lightgbm.dylib, 1): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
[info] Referenced from: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives3454868880605903181/lib_lightgbm.dylib
[info] Reason: image not found
```

**To Reproduce**
`sbt "testOnly com.microsoft.ml.spark.lightgbm.split2.VerifyLightGBMRanker"`

**Expected behavior**
Tests to pass

**Info (please complete the following information):**
- Latest master
- OSX 10.14.6

** Stacktrace**
```
[info] + Test Verify LightGBM Ranker on ranking dataset took 0.198s
19/08/26 13:21:24 ERROR Executor: Exception in task 3.0 in stage 16.0 (TID 49)
java.lang.UnsatisfiedLinkError: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives997331245052213810/lib_lightgbm.dylib: dlopen(/private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives997331245052213810/lib_lightgbm.dylib, 1): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives997331245052213810/lib_lightgbm.dylib
Reason: image not found
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at com.microsoft.ml.spark.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:59)
at com.microsoft.ml.spark.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:40)
at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$15.apply(TrainUtils.scala:404)
at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$15.apply(TrainUtils.scala:401)
at com.microsoft.ml.spark.core.env.StreamUtilities$.using(StreamUtilities.scala:28)
at com.microsoft.ml.spark.lightgbm.TrainUtils$.trainLightGBM(TrainUtils.scala:400)
at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$6.apply(LightGBMBase.scala:128)
at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$6.apply(LightGBMBase.scala:128)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
19/08/26 13:21:24 ERROR Executor: Exception in task 0.0 in stage 16.0 (TID 46)
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with sbt "testOnly com.microsoft.ml.spark.lightgbm.split2.VerifyLightGBMRanker" and start at NativeLoader.loadLibraryByName and LightGBMUtils.scala:40. Trace how the LightGBM native library is loaded on macOS and inspect the related native-library packaging or build configuration. Done means the reported ranking test passes on the stated macOS environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, scala, spark
Domain
machine-learning, operating-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.