awslabs / awslabs/python-deequ

py4j.protocol.Py4JError while adding Analyzer in PyDeequ + PySpark

Offen
#154 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug researching
Vorherrschende Sprache
Jupyter Notebook
Sterne
826
Forks
158
Ø Merge
9 T. 22 Std.
Gemergte PRs (30 T.)
3

Beschreibung

**Describe the bug**
I'm trying to add analysers for the columns of a PySpark Data Frame, I'm getting py4j.protocol.Py4JError.

**To Reproduce**
Steps to reproduce the behavior:
1. Install pydeequ==1.1.0, py4j==0.10.9.7 , pyspark==3.4.1 and Spark == 3.3.0.
2. Read CSV to PySpark DataFrame.
3. Instantiate the Analsysis Runner :: analysis_runner = AnalysisRunner(spark).onData(df).addAnalyzer(Size())
4. Add Completeness and Uniqueness DQ rules for every columns as:

for column in df.columns:
analysis_runner.addAnalyzer(Uniqueness(column))
analysis_runner.addAnalyzer(Completeness(column))
analysisResult= analysis_runner.run()

5. Run The Pyspark script to see following error:

analysis_runner.addAnalyzer(Uniqueness(column))
File "/Users/indusharma/Library/Python/3.9/lib/python/site-packages/pydeequ/analyzers.py", line 134, in addAnalyzer
_analyzer_jvm = analyzer._analyzer_jvm
File "/Users/indusharma/Library/Python/3.9/lib/python/site-packages/pydeequ/analyzers.py", line 775, in _analyzer_jvm
to_scala_seq(self._jvm, self.columns), self._jvm.scala.Option.apply(self.where)
File "/Users/indusharma/Library/Python/3.9/lib/python/site-packages/pydeequ/scala_utils.py", line 80, in to_scala_seq
return jvm.scala.collection.JavaConversions.iterableAsScalaIterable(iterable).toSeq()
File "/Users/indusharma/Library/Python/3.9/lib/python/site-packages/py4j/java_gateway.py", line 1322, in __call__
return_value = get_return_value(
File "/Users/indusharma/Library/Python/3.9/lib/python/site-packages/pyspark/errors/exceptions/captured.py", line 169, in deco
return f(*a, **kw)
File "/Users/indusharma/Library/Python/3.9/lib/python/site-packages/py4j/protocol.py", line 330, in get_return_value
raise Py4JError(
py4j.protocol.Py4JError: An error occurred while calling z:scala.collection.JavaConversions.iterableAsScalaIterable. Trace:
py4j.Py4JException: Method iterableAsScalaIterable([class java.lang.String]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:321)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:342)
at py4j.Gateway.invoke(Gateway.java:276)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)

**Expected behavior**
Analysers are run successfully.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
N/A

**Smartphone (please complete the following information):**
N/A

**Additional context**

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start with pydeequ/analyzers.py at addAnalyzer and the _analyzer_jvm path, then inspect pydeequ/scala_utils.py at to_scala_seq. Reproduce the reported setup with pydeequ 1.1.0, py4j 0.10.9.7, PySpark 3.4.1, and Spark 3.3.0; done means Completeness and Uniqueness analyzers can be added and run successfully.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, spark
Bereich
data-engineering
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.