microsoft / microsoft/SynapseML
java.lang.NoSuchMethodError: spray.json.BasicFormats.$init$(Lspray/json/BasicFormats;)V
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
**Describe the bug**
I am using Databricks DBR 8.3ML and not able to instantiate LightGBMClassifier. Followed the steps in the documentation
**To Reproduce**
Databricks DBR 8.3ML | Spark 3.1.1 | Scala 2.12
SynapseML installed Maven: com.microsoft.azure:synapseml_2.12:0.9.4
**Expected behavior**
Library installed successfully.
Below command executed sucessfully -
from synapse.ml.lightgbm import LightGBMClassifier
**Info (please complete the following information):**
- SynapseML Version: 0.9.4
- Spark Version - 3.1.1
- Spark Platform - Databricks DBR 8.3ML
** Stacktrace**
```
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
in
----> 1 m1 = LightGBMClassifier(objective="binary")
/databricks/spark/python/pyspark/__init__.py in wrapper(self, *args, **kwargs)
112 raise TypeError("Method %s forces keyword arguments." % func.__name__)
113 self._input_kwargs = kwargs
--> 114 return func(self, **kwargs)
115 return wrapper
116
/local_disk0/spark-e1502211-937e-4011-a1c8-38eb6e14ccc1/userFiles-c4de24ed-7b19-4da7-a212-a1877977fb1e/addedFile4981733457150739342synapseml_lightgbm_2_12_0_9_4-a075c.jar/synapse/ml/lightgbm/LightGBMClassifier.py in __init__(self, java_obj, baggingFraction, baggingFreq, baggingSeed, binSampleCount, boostFromAverage, boostingType, categoricalSlotIndexes, categoricalSlotNames, chunkSize, defaultListenPort, driverListenPort, dropRate, earlyStoppingRound, featureFraction, featuresCol, featuresShapCol, fobj, improvementTolerance, initScoreCol, isProvideTrainingMetric, isUnbalance, labelCol, lambdaL1, lambdaL2, leafPredictionCol, learningRate, matrixType, maxBin, maxBinByFeature, maxDeltaStep, maxDepth, maxDrop, metric, minDataInLeaf, minGainToSplit, minSumHessianInLeaf, modelString, negBaggingFraction, numBatches, numIterations, numLeaves, numTasks, numThreads, objective, parallelism, posBaggingFraction, predictionCol, probabilityCol, rawPredictionCol, repartitionByGroupingColumn, skipDrop, slotNames, thresholds, timeout, topK, uniformDrop, useBarrierExecutionMode, useSingleDatasetMode, validationIndicatorCol, verbosity, weightCol, xgboostDartMode)
283 super(LightGBMClassifier, self).__init__()
284 if java_obj is None:
--> 285 self._java_obj = self._new_java_obj("com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier", self.uid)
286 else:
287 self._java_obj = java_obj
/databricks/spark/python/pyspark/ml/wrapper.py in _new_java_obj(java_class, *args)
64 java_obj = getattr(java_obj, name)
65 java_args = [_py2java(sc, arg) for arg in args]
---> 66 return java_obj(*java_args)
67
68 @staticmethod
/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
1566
1567 answer = self._gateway_client.send_command(command)
-> 1568 return_value = get_return_value(
1569 answer, self._gateway_client, None, self._fqn)
1570
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
115 def deco(*a, **kw):
116 try:
--> 117 return f(*a, **kw)
118 except py4j.protocol.Py4JJavaError as e:
119 converted = convert_exception(e.java_exception)
/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling None.com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.
: java.lang.NoSuchMethodError: spray.json.BasicFormats.$init$(Lspray/json/BasicFormats;)V
at com.microsoft.azure.synapse.ml.logging.LogJsonProtocol$.(BasicLogging.scala:18)
at com.microsoft.azure.synapse.ml.logging.LogJsonProtocol$.(BasicLogging.scala)
at com.microsoft.azure.synapse.ml.logging.BasicLogging.logBase(BasicLogging.scala:31)
at com.microsoft.azure.synapse.ml.logging.BasicLogging.logBase$(BasicLogging.scala:30)
at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logBase(LightGBMClassifier.scala:26)
at com.microsoft.azure.synapse.ml.logging.BasicLogging.logClass(BasicLogging.scala:41)
at com.microsoft.azure.synapse.ml.logging.BasicLogging.logClass$(BasicLogging.scala:40)
at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logClass(LightGBMClassifier.scala:26)
at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.(LightGBMClassifier.scala:29)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:250)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
```
If the bug pertains to a specific feature please tag the appropriate [CODEOWNER](https://github.com/Microsoft/SynapseML/blob/master/CODEOWNERS) for better visibility
**Additional context**
Add any other context about the problem here.
AB#1984494
Contributor guide
Research direction
Start by reproducing the LightGBMClassifier construction in the reported Databricks DBR 8.3ML, Spark 3.1.1, Scala 2.12, and SynapseML 0.9.4 environment. Inspect the stacktrace references to BasicLogging.scala:18 and LightGBMClassifier.scala:29, then check the dependency versions involved in spray.json. Done means the classifier instantiates without the NoSuchMethodError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scala
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100