microsoft / microsoft/SynapseML

[BUG] Getting errors while using SynapseML Lambda() transformer

Open
#1,926 5 comments 0 reactions 1 assignee View on GitHub

@mhamilton723 is already working on this.

Since May 1, 2023.

area/lambda bug
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

### SynapseML version

0.11.0

### System information

python = 3.9.16
pyspark = 3.3.0
synapseml = 0.11.0
Environment used: Google colab botebook

### Describe the problem

Getting below error while using SynapseML [lambda()](https://microsoft.github.io/SynapseML/docs/documentation/transformers/transformers_core/#lambda) transformer.

### Code to reproduce issue

from synapse.ml.stages import *
from pyspark.sql.types import StringType, StructType

df = (spark.createDataFrame([
(0, 0.0, "guitars", "drums", 1, True),
(1, 1.0, "piano", "trumpet", 2, False),
(2, 2.0, "bass", "cymbals", 3, True)
], ["numbers", "doubles", "words", "more", "longs", "booleans"]))

def transformFunc(df):
return df.select("numbers")

def transformSchemaFunc(schema):
return StructType([schema("numbers")])

l = (Lambda()
.setTransformFunc(transformFunc)
.setTransformSchemaFunc(transformSchemaFunc))

l.transform(df).show()

### Other info / logs

---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
[](https://localhost:8080/#) in ()
----> 1 l.transform(df).show()

6 frames
[/usr/local/lib/python3.9/dist-packages/py4j/protocol.py](https://localhost:8080/#) in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)

Py4JJavaError: An error occurred while calling o52.w.
: java.lang.ClassCastException: class net.razorvine.pickle.objects.ClassDictConstructor cannot be cast to class org.apache.spark.sql.expressions.UserDefinedFunction (net.razorvine.pickle.objects.ClassDictConstructor and org.apache.spark.sql.expressions.UserDefinedFunction are in unnamed module of loader 'app')
at com.microsoft.azure.synapse.ml.param.UDFParam$$anonfun$$lessinit$greater$1.apply(UDFParam.scala:26)
at org.apache.spark.ml.param.Param.validate(params.scala:70)
at org.apache.spark.ml.param.ParamPair.(params.scala:634)
at org.apache.spark.ml.param.Param.$minus$greater(params.scala:85)
at org.apache.spark.ml.param.Param.w(params.scala:81)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)

### What component(s) does this bug affect?

- [ ] `area/cognitive`: Cognitive project
- [X] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [ ] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue

### What language(s) does this bug affect?

- [ ] `language/scala`: Scala source code
- [ ] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.