microsoft / microsoft/SynapseML

[BUG] LightGBMRegressor : 'JavaPackage' object is not callable

Open
#2,240 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

### SynapseML version

0.11.4

### System information

- **Language version** (e.g. python 3.8, scala 2.12): 3.10.13
- **Spark Version** (e.g. 3.2.3): 3.3.4
- **Spark Platform** (e.g. Synapse, Databricks): Synapse

PySpark version: 3.3.4
Spark version: 3.3.4
SynapseML version: 0.11.4
Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]

Screenshot 2024-06-18 at 12 50 33 PM

### Describe the problem

Trying to get start with SynapseML - But when ever we call model train function.
Getting 'JavaPackage' object is not callable error either in my local mac or remote jupyter server

### Code to reproduce issue

import pyspark
spark = pyspark.sql.SparkSession.builder.appName("MyApp") \
.config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.11.4-spark3.3") \
.config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
.getOrCreate()
import synapse.ml

train, test = (
spark.read.parquet(
"wasbs://publicwasb@mmlspark.blob.core.windows.net/BookReviewsFromAmazon10K.parquet"
)
.limit(1000)
.cache()
.randomSplit([0.8, 0.2])
)

display(train)

from pyspark.ml import Pipeline
from synapse.ml.featurize.text import TextFeaturizer
from synapse.ml.lightgbm import LightGBMRegressor

model = Pipeline(
stages=[
TextFeaturizer(inputCol="text", outputCol="features"),
LightGBMRegressor(featuresCol="features", labelCol="rating"),
]
).fit(train)

### Other info / logs

---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[10], line 7
2 from synapse.ml.featurize.text import TextFeaturizer
3 from synapse.ml.lightgbm import LightGBMRegressor
5 model = Pipeline(
6 stages=[
----> 7 TextFeaturizer(inputCol="text", outputCol="features"),
8 LightGBMRegressor(featuresCol="features", labelCol="rating"),
9 ]
10 ).fit(train)

File /opt/shan/spark/python/lib/pyspark.zip/pyspark/__init__.py:135, in keyword_only..wrapper(self, *args, **kwargs)
133 raise TypeError("Method %s forces keyword arguments." % func.__name__)
134 self._input_kwargs = kwargs
--> 135 return func(self, **kwargs)

File ~/user-libs/python/synapse/ml/featurize/text/TextFeaturizer.py:106, in TextFeaturizer.__init__(self, java_obj, binary, caseSensitiveStopWords, defaultStopWordLanguage, inputCol, minDocFreq, minTokenLength, nGramLength, numFeatures, outputCol, stopWords, toLowercase, tokenizerGaps, tokenizerPattern, useIDF, useNGram, useStopWordsRemover, useTokenizer)
104 super(TextFeaturizer, self).__init__()
105 if java_obj is None:
--> 106 self._java_obj = self._new_java_obj("com.microsoft.azure.synapse.ml.featurize.text.TextFeaturizer", self.uid)
107 else:
108 self._java_obj = java_obj

File /opt/shan/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py:86, in JavaWrapper._new_java_obj(java_class, *args)
84 java_obj = getattr(java_obj, name)
85 java_args = [_py2java(sc, arg) for arg in args]
---> 86 return java_obj(*java_args)

TypeError: 'JavaPackage' object is not callable

### What component(s) does this bug affect?

- [ ] `area/cognitive`: Cognitive project
- [ ] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [X] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [X] `area/models`: models related issue

### What language(s) does this bug affect?

- [ ] `language/scala`: Scala source code
- [X] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [X] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations

Contributor guide

Open the contributing guide

Research direction

Start with the provided SparkSession package configuration and the TextFeaturizer.py constructor, especially the _new_java_obj call shown in the traceback. Reproduce the pipeline initialization with Spark 3.3.4, SynapseML 0.11.4, and Python 3.10.13, then verify that TextFeaturizer and LightGBMRegressor can be constructed and the Pipeline fit completes without the JavaPackage error.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python, scala, spark
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.