awslabs / awslabs/python-deequ

Can't execute ConstraintSuggestionRunner: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist

Open
#70 6 comments 6 reactions 0 assignees View on GitHub
bug question
Dominant language
Jupyter Notebook
Stars
826
Forks
158
PR merge metrics
No merged PRs in 30d

Description

I'm trying to run the ConstraintSuggestionRunner with the latest version of pyDeequ that supports Spark 3.1. I encountered the following error when I was running this code
```
from pydeequ.suggestions import *

suggestionResult = ConstraintSuggestionRunner(spark) \
.onData(df) \
.addConstraintRule(DEFAULT()) \
.run()

print(json.dumps(suggestionResult, indent=2))
```
```
---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
in
1 from pydeequ.suggestions import *
2
----> 3 suggestionResult = ConstraintSuggestionRunner(spark) \
4 .onData(df) \
5 .addConstraintRule(DEFAULT()) \

/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in addConstraintRule(self, constraintRule)
64 for rule in constraintRule_jvm:
65 rule._set_jvm(self._jvm)
---> 66 rule_jvm = rule.rule_jvm
67 self._ConstraintSuggestionRunBuilder.addConstraintRule(rule_jvm)
68

/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in rule_jvm(self)
184 @property
185 def rule_jvm(self):
--> 186 return self._deequSuggestions.rules.CategoricalRangeRule()
187
188

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
1566
1567 answer = self._gateway_client.send_command(command)
-> 1568 return_value = get_return_value(
1569 answer, self._gateway_client, None, self._fqn)
1570

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
108 def deco(*a, **kw):
109 try:
--> 110 return f(*a, **kw)
111 except py4j.protocol.Py4JJavaError as e:
112 converted = convert_exception(e.java_exception)

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
328 format(target_id, ".", name), value)
329 else:
--> 330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
332 format(target_id, ".", name, value))

Py4JError: An error occurred while calling None.com.amazon.deequ.suggestions.rules.CategoricalRangeRule. Trace:
py4j.Py4JException: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:202)
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:219)
at py4j.Gateway.invoke(Gateway.java:248)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)

```

Additional information:
1. I'm working with Spark 3.1, to cooperate with the version, I used the pyDeequ package as instructed [here](https://github.com/awslabs/python-deequ/issues/1). Everything else is exactly the same as written in the [tutorial](https://github.com/awslabs/python-deequ/blob/master/tutorials/suggestions.ipynb)
```
from pyspark.sql import SparkSession, Row, DataFrame
import json
import pandas as pd
import sagemaker_pyspark

import pydeequ

classpath = ":".join(sagemaker_pyspark.classpath_jars())

spark = (SparkSession
.builder
.config("spark.driver.extraClassPath", classpath)
.config("spark.jars.packages", 'deequ-2.0.0-spark-3.1.jar') # this is where i changed
.config("spark.jars.excludes", pydeequ.f2j_maven_coord)
.getOrCreate())
```
2. I tried other functions like VerificationSuite() and AnalysisRunner(), they both work fine.

Is it bc of the version I'm running that haven't supported this specific functionality in Spark 3.1 yet? I think I can avoid this
by downgrading Spark version but I would not do that until no other solutions. Any insights would be really appreciated!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.