[BUG] java.lang.NoSuchMethodError when using synapse.ml.lightgbm

Nobody has claimed this yet.

### SynapseML version

synapseml_2.12:1.0.4

### System information

- **Language version** (e.g. python 3.8, scala 2.12):
- **Spark Version** (e.g. 3.4.1):
- **Spark Platform** ( Databricks):
- ***Databricks Runtime Version*** (13.3 LTS ML (includes Apache Spark 3.4.1, Scala 2.12))

### Describe the problem

I have installed Synapse ML onto my databricks cluster from Maven using cooradinates com.microsoft.azure:synapseml_2.12:1.0.4 which should support spark 3.4.

In my notebook, I have imported the LightGBMRegressor using the code
from synapse.ml.lightgbm.LightGBMRegressor import LightGBMRegressor

But when I call the regressor in the notebook, it throws me an error message

**Error Message:
Error encountered: An error occurred while calling None.com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.
: java.lang.NoSuchMethodError: org.apache.spark.ml.ComplexParamsWritable.$init$(Lorg/apache/spark/ml/ComplexParamsWritable;)V
at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.(LightGBMRegressor.scala:40)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397)
at py4j.Gateway.invoke(Gateway.java:257)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199)
at py4j.ClientServerConnection.run(ClientServerConnection.java:119)
at java.lang.Thread.run(Thread.java:750)**

This is happening because of Spark version mismatch but SynapseML package I installed is expected to support Spark cluster has at least Spark 3.2 and Scala 2.12. This according to theinstallation page of SynapseML. https://microsoft.github.io/SynapseML/docs/Get%20Started/Install%20SynapseML/.

**Kindly help me how to resolve it?**

### Code to reproduce issue

try:
from synapse.ml.lightgbm.LightGBMRegressor import LightGBMRegressor
lgbm_cv = LightGBMRegressor()
except Exception as e:
print(f"Error encountered: {e}")

### Other info / logs

_No response_

### What component(s) does this bug affect?

- [ ] `area/cognitive`: Cognitive project
- [ ] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [ ] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue

### What language(s) does this bug affect?

- [ ] `language/scala`: Scala source code
- [x] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations

Start by reproducing the reported Python snippet with Maven coordinate com.microsoft.azure:synapseml_2.12:1.0.4 on the stated Databricks Runtime. Compare the package's Spark and Scala compatibility with the runtime and trace the LightGBMRegressor constructor failure. Done means the reported regressor can be instantiated without the NoSuchMethodError under a documented supported configuration.

Written by the indexing model from the issue text.