awslabs / awslabs/python-deequ
Can't execute ConstraintSuggestionRunner: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- PR merge metrics
- No merged PRs in 30d
Description
I'm trying to run the ConstraintSuggestionRunner with the latest version of pyDeequ that supports Spark 3.1. I encountered the following error when I was running this code
```
from pydeequ.suggestions import *
suggestionResult = ConstraintSuggestionRunner(spark) \
.onData(df) \
.addConstraintRule(DEFAULT()) \
.run()
print(json.dumps(suggestionResult, indent=2))
```
```
---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
in
1 from pydeequ.suggestions import *
2
----> 3 suggestionResult = ConstraintSuggestionRunner(spark) \
4 .onData(df) \
5 .addConstraintRule(DEFAULT()) \
/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in addConstraintRule(self, constraintRule)
64 for rule in constraintRule_jvm:
65 rule._set_jvm(self._jvm)
---> 66 rule_jvm = rule.rule_jvm
67 self._ConstraintSuggestionRunBuilder.addConstraintRule(rule_jvm)
68
/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in rule_jvm(self)
184 @property
185 def rule_jvm(self):
--> 186 return self._deequSuggestions.rules.CategoricalRangeRule()
187
188
/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
1566
1567 answer = self._gateway_client.send_command(command)
-> 1568 return_value = get_return_value(
1569 answer, self._gateway_client, None, self._fqn)
1570
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
108 def deco(*a, **kw):
109 try:
--> 110 return f(*a, **kw)
111 except py4j.protocol.Py4JJavaError as e:
112 converted = convert_exception(e.java_exception)
/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
328 format(target_id, ".", name), value)
329 else:
--> 330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
332 format(target_id, ".", name, value))
Py4JError: An error occurred while calling None.com.amazon.deequ.suggestions.rules.CategoricalRangeRule. Trace:
py4j.Py4JException: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:202)
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:219)
at py4j.Gateway.invoke(Gateway.java:248)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
```
Additional information:
1. I'm working with Spark 3.1, to cooperate with the version, I used the pyDeequ package as instructed [here](https://github.com/awslabs/python-deequ/issues/1). Everything else is exactly the same as written in the [tutorial](https://github.com/awslabs/python-deequ/blob/master/tutorials/suggestions.ipynb)
```
from pyspark.sql import SparkSession, Row, DataFrame
import json
import pandas as pd
import sagemaker_pyspark
import pydeequ
classpath = ":".join(sagemaker_pyspark.classpath_jars())
spark = (SparkSession
.builder
.config("spark.driver.extraClassPath", classpath)
.config("spark.jars.packages", 'deequ-2.0.0-spark-3.1.jar') # this is where i changed
.config("spark.jars.excludes", pydeequ.f2j_maven_coord)
.getOrCreate())
```
2. I tried other functions like VerificationSuite() and AnalysisRunner(), they both work fine.
Is it bc of the version I'm running that haven't supported this specific functionality in Spark 3.1 yet? I think I can avoid this
by downgrading Spark version but I would not do that until no other solutions. Any insights would be really appreciated!
Contributor guide
Assessment
This issue has not been assessed yet.