awslabs / awslabs/python-deequ

Getting error while running pydeeque locally on spark

Abierto
#35 12 comentarios 3 reacciones 0 asignados Ver en GitHub
documentation question stale
Lenguaje dominante
Jupyter Notebook
Estrellas
826
Forks
158
Merge medio
9 d 22 h
PR fusionados (30 d)
3

Descripción

I am trying to import pydeeque and while running locally on window machine , I am getting following error .

answer = 'xro55', gateway_client = , target_id = None, name = 'com.amazon.deequ.analyzers.Completeness'

def get_return_value(answer, gateway_client, target_id=None, name=None):
"""Converts an answer received from the Java gateway into a Python object.

For example, string representation of integers are converted to Python
integer, string representation of objects are converted to JavaObject
instances, etc.

:param answer: the string returned by the Java gateway
:param gateway_client: the gateway client used to communicate with the Java
Gateway. Only necessary if the answer is a reference (e.g., object,
list, map)
:param target_id: the name of the object from which the answer comes from
(e.g., *object1* in `object1.hello()`). Optional.
:param name: the name of the member from which the answer comes from
(e.g., *hello* in `object1.hello()`). Optional.
"""
if is_error(answer)[0]:
if len(answer) > 1:
type = answer[1]
value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
if answer[1] == REFERENCE_TYPE:
raise Py4JJavaError(
"An error occurred while calling {0}{1}{2}.\n".
> format(target_id, ".", name), value)
E py4j.protocol.Py4JJavaError: An error occurred while calling None.com.amazon.deequ.analyzers.Completeness.
E : java.lang.NoClassDefFoundError: scala/Product$class
E at com.amazon.deequ.analyzers.Completeness.(Completeness.scala:27)
E at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
E at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
E at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
E at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
E at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
E at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
E at py4j.Gateway.invoke(Gateway.java:238)
E at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
E at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
E at py4j.GatewayConnection.run(GatewayConnection.java:238)
E at java.lang.Thread.run(Thread.java:748)
E Caused by: java.lang.ClassNotFoundException: scala.Product$class
E at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
E at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
E at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
E at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
E ... 12 more

C:\SPARK\spark-2.4.2-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py:328: Py4JJavaError

Here are my liberary versions
deequ-1.0.2.jar
pyspark 2.4.7

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Reproduce el fallo en Windows con deequ-1.0.2 y pyspark 2.4.7, empezando por la llamada de Py4J a com.amazon.deequ.analyzers.Completeness. Comprueba la compatibilidad de las dependencias de Spark y Scala indicada por scala/Product$class y la ClassNotFoundException. Se considera terminado cuando pydeeque se pueda importar localmente y se pueda construir el analizador Completeness sin este error.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python, spark
Área
data-engineering
Tipo de issue
Error
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
48/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.