microsoft / microsoft/SynapseML
[BUG] LightGBMRegressor : 'JavaPackage' object is not callable
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
### SynapseML version
0.11.4
### System information
- **Language version** (e.g. python 3.8, scala 2.12): 3.10.13
- **Spark Version** (e.g. 3.2.3): 3.3.4
- **Spark Platform** (e.g. Synapse, Databricks): Synapse
PySpark version: 3.3.4
Spark version: 3.3.4
SynapseML version: 0.11.4
Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]
### Describe the problem
Trying to get start with SynapseML - But when ever we call model train function.
Getting 'JavaPackage' object is not callable error either in my local mac or remote jupyter server
### Code to reproduce issue
import pyspark
spark = pyspark.sql.SparkSession.builder.appName("MyApp") \
.config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.11.4-spark3.3") \
.config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
.getOrCreate()
import synapse.ml
train, test = (
spark.read.parquet(
"wasbs://publicwasb@mmlspark.blob.core.windows.net/BookReviewsFromAmazon10K.parquet"
)
.limit(1000)
.cache()
.randomSplit([0.8, 0.2])
)
display(train)
from pyspark.ml import Pipeline
from synapse.ml.featurize.text import TextFeaturizer
from synapse.ml.lightgbm import LightGBMRegressor
model = Pipeline(
stages=[
TextFeaturizer(inputCol="text", outputCol="features"),
LightGBMRegressor(featuresCol="features", labelCol="rating"),
]
).fit(train)
### Other info / logs
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[10], line 7
2 from synapse.ml.featurize.text import TextFeaturizer
3 from synapse.ml.lightgbm import LightGBMRegressor
5 model = Pipeline(
6 stages=[
----> 7 TextFeaturizer(inputCol="text", outputCol="features"),
8 LightGBMRegressor(featuresCol="features", labelCol="rating"),
9 ]
10 ).fit(train)
File /opt/shan/spark/python/lib/pyspark.zip/pyspark/__init__.py:135, in keyword_only..wrapper(self, *args, **kwargs)
133 raise TypeError("Method %s forces keyword arguments." % func.__name__)
134 self._input_kwargs = kwargs
--> 135 return func(self, **kwargs)
File ~/user-libs/python/synapse/ml/featurize/text/TextFeaturizer.py:106, in TextFeaturizer.__init__(self, java_obj, binary, caseSensitiveStopWords, defaultStopWordLanguage, inputCol, minDocFreq, minTokenLength, nGramLength, numFeatures, outputCol, stopWords, toLowercase, tokenizerGaps, tokenizerPattern, useIDF, useNGram, useStopWordsRemover, useTokenizer)
104 super(TextFeaturizer, self).__init__()
105 if java_obj is None:
--> 106 self._java_obj = self._new_java_obj("com.microsoft.azure.synapse.ml.featurize.text.TextFeaturizer", self.uid)
107 else:
108 self._java_obj = java_obj
File /opt/shan/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py:86, in JavaWrapper._new_java_obj(java_class, *args)
84 java_obj = getattr(java_obj, name)
85 java_args = [_py2java(sc, arg) for arg in args]
---> 86 return java_obj(*java_args)
TypeError: 'JavaPackage' object is not callable
### What component(s) does this bug affect?
- [ ] `area/cognitive`: Cognitive project
- [ ] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [X] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [X] `area/models`: models related issue
### What language(s) does this bug affect?
- [ ] `language/scala`: Scala source code
- [X] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages
### What integration(s) does this bug affect?
- [X] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations
Contributor guide
Research direction
Start with the provided SparkSession package configuration and the TextFeaturizer.py constructor, especially the _new_java_obj call shown in the traceback. Reproduce the pipeline initialization with Spark 3.3.4, SynapseML 0.11.4, and Python 3.10.13, then verify that TextFeaturizer and LightGBMRegressor can be constructed and the Pipeline fit completes without the JavaPackage error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python, scala, spark
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100