microsoft / microsoft/SynapseML

[BUG] Fail to run the Python built from source

Open
#2,350 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug triage
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

### SynapseML version

'0.0.0-1659-1b5df703-SNAPSHOT'

### System information

- **Language version** (e.g. python 3.8, scala 2.12): Python 3.11, Scala version 2.12.17
- **Spark Version** (e.g. 3.2.3): 3.4.1
- **Spark Platform** (e.g. Synapse, Databricks): PySpark

### Describe the problem

When following https://microsoft.github.io/SynapseML/docs/Reference/Developer%20Setup/ to build SynapseML from source, "'JavaPackage' object is not callable" is reported, and if manually copy jars generated in target/scala-2.12 to site-packages/pyspark/jars, "py4j.protocol.Py4JNetworkError: Answer from Java side is empty" would be reported.

### Code to reproduce issue

```python
from synapse.ml.lightgbm import *
LightGBMRegressor()
```

### Other info / logs

```python-traceback
Exception in thread "Thread-26" java.lang.NoClassDefFoundError: spray/json/JsonFormat
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at py4j.reflection.CurrentThreadClassLoadingStrategy.classForName(CurrentThreadClassLoadingStrategy.java:40)
at py4j.reflection.ReflectionUtil.classForName(ReflectionUtil.java:51)
at py4j.reflection.TypeUtil.forName(TypeUtil.java:243)
at py4j.commands.ReflectionCommand.getUnknownMember(ReflectionCommand.java:175)
at py4j.commands.ReflectionCommand.execute(ReflectionCommand.java:87)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: spray.json.JsonFormat
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
... 10 more
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/home/luyanaa/miniforge3/envs/synapseml/lib/python3.11/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 516, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/luyanaa/miniforge3/envs/synapseml/lib/python3.11/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1038, in send_command
response = connection.send_command(command)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/luyanaa/miniforge3/envs/synapseml/lib/python3.11/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 539, in send_command
raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving
Traceback (most recent call last):
File "", line 1, in
File "/home/luyanaa/miniforge3/envs/synapseml/lib/python3.11/site-packages/pyspark/__init__.py", line 139, in wrapper
return func(self, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/home/luyanaa/miniforge3/envs/synapseml/lib/python3.11/site-packages/synapse/ml/featurize/text/TextFeaturizer.py", line 106, in __init__
self._java_obj = self._new_java_obj("com.microsoft.azure.synapse.ml.featurize.text.TextFeaturizer", self.uid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/luyanaa/miniforge3/envs/synapseml/lib/python3.11/site-packages/pyspark/ml/wrapper.py", line 84, in _new_java_obj
java_obj = getattr(java_obj, name)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/luyanaa/miniforge3/envs/synapseml/lib/python3.11/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1664, in __getattr__
py4j.protocol.Py4JError: com.microsoft.azure.synapse.ml.featurize.text.TextFeaturizer does not exist in the JVM
```

### What component(s) does this bug affect?

- [ ] `area/cognitive`: Cognitive project
- [x] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [x] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue

### What language(s) does this bug affect?

- [ ] `language/scala`: Scala source code
- [x] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Follow the Developer Setup instructions and reproduce the failure with Python 3.11, Spark 3.4.1, and the LightGBM example. Inspect the source-built jars under target/scala-2.12 and the PySpark jar loading path, focusing on the missing spray/json dependency shown in the logs. Done means the documented source-build setup can instantiate the reported Python API without JavaPackage or Py4J errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scala, spark
Domain
backend, build-system, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.