microsoft / microsoft/SynapseML

java.lang.NoSuchMethodError: spray.json.BasicFormats.$init$(Lspray/json/BasicFormats;)V

Open
#1,302 1 comment 0 reactions 0 assignees View on GitHub
area/lightgbm
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

**Describe the bug**
I am using Databricks DBR 8.3ML and not able to instantiate LightGBMClassifier. Followed the steps in the documentation

**To Reproduce**
Databricks DBR 8.3ML | Spark 3.1.1 | Scala 2.12
SynapseML installed Maven: com.microsoft.azure:synapseml_2.12:0.9.4

**Expected behavior**
Library installed successfully.
Below command executed sucessfully -
from synapse.ml.lightgbm import LightGBMClassifier

**Info (please complete the following information):**
- SynapseML Version: 0.9.4
- Spark Version - 3.1.1
- Spark Platform - Databricks DBR 8.3ML

** Stacktrace**
```
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
in
----> 1 m1 = LightGBMClassifier(objective="binary")

/databricks/spark/python/pyspark/__init__.py in wrapper(self, *args, **kwargs)
112 raise TypeError("Method %s forces keyword arguments." % func.__name__)
113 self._input_kwargs = kwargs
--> 114 return func(self, **kwargs)
115 return wrapper
116

/local_disk0/spark-e1502211-937e-4011-a1c8-38eb6e14ccc1/userFiles-c4de24ed-7b19-4da7-a212-a1877977fb1e/addedFile4981733457150739342synapseml_lightgbm_2_12_0_9_4-a075c.jar/synapse/ml/lightgbm/LightGBMClassifier.py in __init__(self, java_obj, baggingFraction, baggingFreq, baggingSeed, binSampleCount, boostFromAverage, boostingType, categoricalSlotIndexes, categoricalSlotNames, chunkSize, defaultListenPort, driverListenPort, dropRate, earlyStoppingRound, featureFraction, featuresCol, featuresShapCol, fobj, improvementTolerance, initScoreCol, isProvideTrainingMetric, isUnbalance, labelCol, lambdaL1, lambdaL2, leafPredictionCol, learningRate, matrixType, maxBin, maxBinByFeature, maxDeltaStep, maxDepth, maxDrop, metric, minDataInLeaf, minGainToSplit, minSumHessianInLeaf, modelString, negBaggingFraction, numBatches, numIterations, numLeaves, numTasks, numThreads, objective, parallelism, posBaggingFraction, predictionCol, probabilityCol, rawPredictionCol, repartitionByGroupingColumn, skipDrop, slotNames, thresholds, timeout, topK, uniformDrop, useBarrierExecutionMode, useSingleDatasetMode, validationIndicatorCol, verbosity, weightCol, xgboostDartMode)
283 super(LightGBMClassifier, self).__init__()
284 if java_obj is None:
--> 285 self._java_obj = self._new_java_obj("com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier", self.uid)
286 else:
287 self._java_obj = java_obj

/databricks/spark/python/pyspark/ml/wrapper.py in _new_java_obj(java_class, *args)
64 java_obj = getattr(java_obj, name)
65 java_args = [_py2java(sc, arg) for arg in args]
---> 66 return java_obj(*java_args)
67
68 @staticmethod

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
1566
1567 answer = self._gateway_client.send_command(command)
-> 1568 return_value = get_return_value(
1569 answer, self._gateway_client, None, self._fqn)
1570

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
115 def deco(*a, **kw):
116 try:
--> 117 return f(*a, **kw)
118 except py4j.protocol.Py4JJavaError as e:
119 converted = convert_exception(e.java_exception)

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)

Py4JJavaError: An error occurred while calling None.com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.
: java.lang.NoSuchMethodError: spray.json.BasicFormats.$init$(Lspray/json/BasicFormats;)V
at com.microsoft.azure.synapse.ml.logging.LogJsonProtocol$.(BasicLogging.scala:18)
at com.microsoft.azure.synapse.ml.logging.LogJsonProtocol$.(BasicLogging.scala)
at com.microsoft.azure.synapse.ml.logging.BasicLogging.logBase(BasicLogging.scala:31)
at com.microsoft.azure.synapse.ml.logging.BasicLogging.logBase$(BasicLogging.scala:30)
at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logBase(LightGBMClassifier.scala:26)
at com.microsoft.azure.synapse.ml.logging.BasicLogging.logClass(BasicLogging.scala:41)
at com.microsoft.azure.synapse.ml.logging.BasicLogging.logClass$(BasicLogging.scala:40)
at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logClass(LightGBMClassifier.scala:26)
at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.(LightGBMClassifier.scala:29)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:250)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
```

If the bug pertains to a specific feature please tag the appropriate [CODEOWNER](https://github.com/Microsoft/SynapseML/blob/master/CODEOWNERS) for better visibility

**Additional context**
Add any other context about the problem here.

AB#1984494

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the LightGBMClassifier construction in the reported Databricks DBR 8.3ML, Spark 3.1.1, Scala 2.12, and SynapseML 0.9.4 environment. Inspect the stacktrace references to BasicLogging.scala:18 and LightGBMClassifier.scala:29, then check the dependency versions involved in spray.json. Done means the classifier instantiates without the NoSuchMethodError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scala
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.