microsoft / microsoft/SynapseML
[BUG] Getting errors while using SynapseML Lambda() transformer
@mhamilton723 is already working on this.
Since May 1, 2023.
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
### SynapseML version
0.11.0
### System information
python = 3.9.16
pyspark = 3.3.0
synapseml = 0.11.0
Environment used: Google colab botebook
### Describe the problem
Getting below error while using SynapseML [lambda()](https://microsoft.github.io/SynapseML/docs/documentation/transformers/transformers_core/#lambda) transformer.
### Code to reproduce issue
from synapse.ml.stages import *
from pyspark.sql.types import StringType, StructType
df = (spark.createDataFrame([
(0, 0.0, "guitars", "drums", 1, True),
(1, 1.0, "piano", "trumpet", 2, False),
(2, 2.0, "bass", "cymbals", 3, True)
], ["numbers", "doubles", "words", "more", "longs", "booleans"]))
def transformFunc(df):
return df.select("numbers")
def transformSchemaFunc(schema):
return StructType([schema("numbers")])
l = (Lambda()
.setTransformFunc(transformFunc)
.setTransformSchemaFunc(transformSchemaFunc))
l.transform(df).show()
### Other info / logs
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
[](https://localhost:8080/#) in ()
----> 1 l.transform(df).show()
6 frames
[/usr/local/lib/python3.9/dist-packages/py4j/protocol.py](https://localhost:8080/#) in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling o52.w.
: java.lang.ClassCastException: class net.razorvine.pickle.objects.ClassDictConstructor cannot be cast to class org.apache.spark.sql.expressions.UserDefinedFunction (net.razorvine.pickle.objects.ClassDictConstructor and org.apache.spark.sql.expressions.UserDefinedFunction are in unnamed module of loader 'app')
at com.microsoft.azure.synapse.ml.param.UDFParam$$anonfun$$lessinit$greater$1.apply(UDFParam.scala:26)
at org.apache.spark.ml.param.Param.validate(params.scala:70)
at org.apache.spark.ml.param.ParamPair.(params.scala:634)
at org.apache.spark.ml.param.Param.$minus$greater(params.scala:85)
at org.apache.spark.ml.param.Param.w(params.scala:81)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
### What component(s) does this bug affect?
- [ ] `area/cognitive`: Cognitive project
- [X] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [ ] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue
### What language(s) does this bug affect?
- [ ] `language/scala`: Scala source code
- [ ] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages
### What integration(s) does this bug affect?
- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.