awslabs / awslabs/python-deequ

Can't execute ConstraintSuggestionRunner: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist

Abierto
#70 6 comentarios 6 reacciones 0 asignados Ver en GitHub
bug question
Lenguaje dominante
Jupyter Notebook
Estrellas
826
Forks
158
Merge medio
9 d 22 h
PR fusionados (30 d)
3

Descripción

I'm trying to run the ConstraintSuggestionRunner with the latest version of pyDeequ that supports Spark 3.1. I encountered the following error when I was running this code
```
from pydeequ.suggestions import *

suggestionResult = ConstraintSuggestionRunner(spark) \
.onData(df) \
.addConstraintRule(DEFAULT()) \
.run()

print(json.dumps(suggestionResult, indent=2))
```
```
---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
in
1 from pydeequ.suggestions import *
2
----> 3 suggestionResult = ConstraintSuggestionRunner(spark) \
4 .onData(df) \
5 .addConstraintRule(DEFAULT()) \

/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in addConstraintRule(self, constraintRule)
64 for rule in constraintRule_jvm:
65 rule._set_jvm(self._jvm)
---> 66 rule_jvm = rule.rule_jvm
67 self._ConstraintSuggestionRunBuilder.addConstraintRule(rule_jvm)
68

/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in rule_jvm(self)
184 @property
185 def rule_jvm(self):
--> 186 return self._deequSuggestions.rules.CategoricalRangeRule()
187
188

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
1566
1567 answer = self._gateway_client.send_command(command)
-> 1568 return_value = get_return_value(
1569 answer, self._gateway_client, None, self._fqn)
1570

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
108 def deco(*a, **kw):
109 try:
--> 110 return f(*a, **kw)
111 except py4j.protocol.Py4JJavaError as e:
112 converted = convert_exception(e.java_exception)

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
328 format(target_id, ".", name), value)
329 else:
--> 330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
332 format(target_id, ".", name, value))

Py4JError: An error occurred while calling None.com.amazon.deequ.suggestions.rules.CategoricalRangeRule. Trace:
py4j.Py4JException: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:202)
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:219)
at py4j.Gateway.invoke(Gateway.java:248)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)

```

Additional information:
1. I'm working with Spark 3.1, to cooperate with the version, I used the pyDeequ package as instructed [here](https://github.com/awslabs/python-deequ/issues/1). Everything else is exactly the same as written in the [tutorial](https://github.com/awslabs/python-deequ/blob/master/tutorials/suggestions.ipynb)
```
from pyspark.sql import SparkSession, Row, DataFrame
import json
import pandas as pd
import sagemaker_pyspark

import pydeequ

classpath = ":".join(sagemaker_pyspark.classpath_jars())

spark = (SparkSession
.builder
.config("spark.driver.extraClassPath", classpath)
.config("spark.jars.packages", 'deequ-2.0.0-spark-3.1.jar') # this is where i changed
.config("spark.jars.excludes", pydeequ.f2j_maven_coord)
.getOrCreate())
```
2. I tried other functions like VerificationSuite() and AnalysisRunner(), they both work fine.

Is it bc of the version I'm running that haven't supported this specific functionality in Spark 3.1 yet? I think I can avoid this
by downgrading Spark version but I would not do that until no other solutions. Any insights would be really appreciated!

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza en pydeequ/suggestions.py, en el método rule_jvm mostrado en el traceback, y luego inspecciona el constructor de CategoricalRangeRule expuesto por deequ-2.0.0-spark-3.1.jar. Reproduce el tutorial de suggestions con Spark 3.1 y verifica que DEFAULT() se ejecute sin el error de constructor de Py4J.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java, python
Área
data-engineering
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.