awslabs / awslabs/python-deequ

Can't execute ConstraintSuggestionRunner: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist

オープン
#70 コメント 6 件 リアクション 6 件 担当者 0 名 GitHub で見る
bug question
主要言語
Jupyter Notebook
スター
826
フォーク
158
平均マージ
9日 22時間
マージ済み PR(30日)
3

説明

I'm trying to run the ConstraintSuggestionRunner with the latest version of pyDeequ that supports Spark 3.1. I encountered the following error when I was running this code
```
from pydeequ.suggestions import *

suggestionResult = ConstraintSuggestionRunner(spark) \
.onData(df) \
.addConstraintRule(DEFAULT()) \
.run()

print(json.dumps(suggestionResult, indent=2))
```
```
---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
in
1 from pydeequ.suggestions import *
2
----> 3 suggestionResult = ConstraintSuggestionRunner(spark) \
4 .onData(df) \
5 .addConstraintRule(DEFAULT()) \

/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in addConstraintRule(self, constraintRule)
64 for rule in constraintRule_jvm:
65 rule._set_jvm(self._jvm)
---> 66 rule_jvm = rule.rule_jvm
67 self._ConstraintSuggestionRunBuilder.addConstraintRule(rule_jvm)
68

/local_disk0/.ephemeral_nfs/envs/pythonEnv-5e8d3820-b76d-4b1b-ab52-2e507e080d3f/lib/python3.8/site-packages/pydeequ/suggestions.py in rule_jvm(self)
184 @property
185 def rule_jvm(self):
--> 186 return self._deequSuggestions.rules.CategoricalRangeRule()
187
188

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
1566
1567 answer = self._gateway_client.send_command(command)
-> 1568 return_value = get_return_value(
1569 answer, self._gateway_client, None, self._fqn)
1570

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
108 def deco(*a, **kw):
109 try:
--> 110 return f(*a, **kw)
111 except py4j.protocol.Py4JJavaError as e:
112 converted = convert_exception(e.java_exception)

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
328 format(target_id, ".", name), value)
329 else:
--> 330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
332 format(target_id, ".", name, value))

Py4JError: An error occurred while calling None.com.amazon.deequ.suggestions.rules.CategoricalRangeRule. Trace:
py4j.Py4JException: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:202)
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:219)
at py4j.Gateway.invoke(Gateway.java:248)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)

```

Additional information:
1. I'm working with Spark 3.1, to cooperate with the version, I used the pyDeequ package as instructed [here](https://github.com/awslabs/python-deequ/issues/1). Everything else is exactly the same as written in the [tutorial](https://github.com/awslabs/python-deequ/blob/master/tutorials/suggestions.ipynb)
```
from pyspark.sql import SparkSession, Row, DataFrame
import json
import pandas as pd
import sagemaker_pyspark

import pydeequ

classpath = ":".join(sagemaker_pyspark.classpath_jars())

spark = (SparkSession
.builder
.config("spark.driver.extraClassPath", classpath)
.config("spark.jars.packages", 'deequ-2.0.0-spark-3.1.jar') # this is where i changed
.config("spark.jars.excludes", pydeequ.f2j_maven_coord)
.getOrCreate())
```
2. I tried other functions like VerificationSuite() and AnalysisRunner(), they both work fine.

Is it bc of the version I'm running that haven't supported this specific functionality in Spark 3.1 yet? I think I can avoid this
by downgrading Spark version but I would not do that until no other solutions. Any insights would be really appreciated!

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

traceback に示されている rule_jvm メソッドがある pydeequ/suggestions.py から始め、次に deequ-2.0.0-spark-3.1.jar が公開している CategoricalRangeRule コンストラクターを調べます。Spark 3.1 で suggestions チュートリアルを再現し、DEFAULT() が Py4J コンストラクターエラーなしで実行されることを確認します。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java, python
領域
data-engineering
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
静か
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。