awslabs / awslabs/python-deequ
Getting error while running pydeeque locally on spark
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- PR merge metrics
- No merged PRs in 30d
Description
I am trying to import pydeeque and while running locally on window machine , I am getting following error .
answer = 'xro55', gateway_client = , target_id = None, name = 'com.amazon.deequ.analyzers.Completeness'
def get_return_value(answer, gateway_client, target_id=None, name=None):
"""Converts an answer received from the Java gateway into a Python object.
For example, string representation of integers are converted to Python
integer, string representation of objects are converted to JavaObject
instances, etc.
:param answer: the string returned by the Java gateway
:param gateway_client: the gateway client used to communicate with the Java
Gateway. Only necessary if the answer is a reference (e.g., object,
list, map)
:param target_id: the name of the object from which the answer comes from
(e.g., *object1* in `object1.hello()`). Optional.
:param name: the name of the member from which the answer comes from
(e.g., *hello* in `object1.hello()`). Optional.
"""
if is_error(answer)[0]:
if len(answer) > 1:
type = answer[1]
value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
if answer[1] == REFERENCE_TYPE:
raise Py4JJavaError(
"An error occurred while calling {0}{1}{2}.\n".
> format(target_id, ".", name), value)
E py4j.protocol.Py4JJavaError: An error occurred while calling None.com.amazon.deequ.analyzers.Completeness.
E : java.lang.NoClassDefFoundError: scala/Product$class
E at com.amazon.deequ.analyzers.Completeness.(Completeness.scala:27)
E at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
E at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
E at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
E at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
E at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
E at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
E at py4j.Gateway.invoke(Gateway.java:238)
E at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
E at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
E at py4j.GatewayConnection.run(GatewayConnection.java:238)
E at java.lang.Thread.run(Thread.java:748)
E Caused by: java.lang.ClassNotFoundException: scala.Product$class
E at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
E at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
E at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
E at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
E ... 12 more
C:\SPARK\spark-2.4.2-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py:328: Py4JJavaError
Here are my liberary versions
deequ-1.0.2.jar
pyspark 2.4.7
Contributor guide
Assessment
This issue has not been assessed yet.