awslabs / awslabs/python-deequ

Can't execute ConstraintSuggestionRunner: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist

Offen
#70 6 Kommentare 6 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug question
Vorherrschende Sprache
Jupyter Notebook
Sterne
826
Forks
158
Ø Merge
9 T. 22 Std.
Gemergte PRs (30 T.)
3

Beschreibung

I'm trying to run the ConstraintSuggestionRunner with the latest version of pyDeequ that supports Spark 3.1. I encountered the following error when I was running this code
```
from pydeequ.suggestions import *

suggestionResult = ConstraintSuggestionRunner(spark) \
.onData(df) \
.addConstraintRule(DEFAULT()) \
.run()

print(json.dumps(suggestionResult, indent=2))
```
```
---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
in
1 from pydeequ.suggestions import *
2
----> 3 suggestionResult = ConstraintSuggestionRunner(spark) \
4 .onData(df) \
5 .addConstraintRule(DEFAULT()) \

/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in addConstraintRule(self, constraintRule)
64 for rule in constraintRule_jvm:
65 rule._set_jvm(self._jvm)
---> 66 rule_jvm = rule.rule_jvm
67 self._ConstraintSuggestionRunBuilder.addConstraintRule(rule_jvm)
68

/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in rule_jvm(self)
184 @property
185 def rule_jvm(self):
--> 186 return self._deequSuggestions.rules.CategoricalRangeRule()
187
188

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
1566
1567 answer = self._gateway_client.send_command(command)
-> 1568 return_value = get_return_value(
1569 answer, self._gateway_client, None, self._fqn)
1570

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
108 def deco(*a, **kw):
109 try:
--> 110 return f(*a, **kw)
111 except py4j.protocol.Py4JJavaError as e:
112 converted = convert_exception(e.java_exception)

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
328 format(target_id, ".", name), value)
329 else:
--> 330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
332 format(target_id, ".", name, value))

Py4JError: An error occurred while calling None.com.amazon.deequ.suggestions.rules.CategoricalRangeRule. Trace:
py4j.Py4JException: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:202)
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:219)
at py4j.Gateway.invoke(Gateway.java:248)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)

```

Additional information:
1. I'm working with Spark 3.1, to cooperate with the version, I used the pyDeequ package as instructed [here](https://github.com/awslabs/python-deequ/issues/1). Everything else is exactly the same as written in the [tutorial](https://github.com/awslabs/python-deequ/blob/master/tutorials/suggestions.ipynb)
```
from pyspark.sql import SparkSession, Row, DataFrame
import json
import pandas as pd
import sagemaker_pyspark

import pydeequ

classpath = ":".join(sagemaker_pyspark.classpath_jars())

spark = (SparkSession
.builder
.config("spark.driver.extraClassPath", classpath)
.config("spark.jars.packages", 'deequ-2.0.0-spark-3.1.jar') # this is where i changed
.config("spark.jars.excludes", pydeequ.f2j_maven_coord)
.getOrCreate())
```
2. I tried other functions like VerificationSuite() and AnalysisRunner(), they both work fine.

Is it bc of the version I'm running that haven't supported this specific functionality in Spark 3.1 yet? I think I can avoid this
by downgrading Spark version but I would not do that until no other solutions. Any insights would be really appreciated!

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginnen Sie in pydeequ/suggestions.py bei der in der Traceback-Ausgabe gezeigten Methode rule_jvm und untersuchen Sie anschließend den von deequ-2.0.0-spark-3.1.jar bereitgestellten Konstruktor von CategoricalRangeRule. Reproduzieren Sie das Suggestions-Tutorial mit Spark 3.1 und überprüfen Sie, dass DEFAULT() ohne den Py4J-Konstruktorfehler ausgeführt wird.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java, python
Bereich
data-engineering
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.