awslabs / awslabs/python-deequ
NoSuchMethodError when attempting to set up analyzer
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Following the Jupyter tutorial at https://github.com/awslabs/python-deequ/blob/master/tutorials/test_data_quality_at_scale.ipynb, I receive the following exception when attempting to set up the analyzer
```
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
/tmp/ipykernel_2741/3771477838.py in
3 analysisResult = AnalysisRunner(spark) \
4 .onData(df) \
----> 5 .addAnalyzer(Size()) \
6 .addAnalyzer(Completeness("review_id")) \
7 .addAnalyzer(ApproxCountDistinct("review_id")) \
~/.local/share/virtualenvs/pydeequ/lib/python3.7/site-packages/pydeequ/analyzers.py in addAnalyzer(self, analyzer)
132 """
133 analyzer._set_jvm(self._jvm)
--> 134 _analyzer_jvm = analyzer._analyzer_jvm
135 self._AnalysisRunBuilder.addAnalyzer(_analyzer_jvm)
136 return self
~/.local/share/virtualenvs/pydeequ/lib/python3.7/site-packages/pydeequ/analyzers.py in _analyzer_jvm(self)
704 :return self
705 """
--> 706 return self._deequAnalyzers.Size(self._jvm.scala.Option.apply(self.where))
707
708
~/.local/share/virtualenvs/pydeequ/lib/python3.7/site-packages/py4j/java_gateway.py in __call__(self, *args)
1523 answer = self._gateway_client.send_command(command)
1524 return_value = get_return_value(
-> 1525 answer, self._gateway_client, None, self._fqn)
1526
1527 for temp_arg in temp_args:
~/.local/share/virtualenvs/pydeequ/lib/python3.7/site-packages/pyspark/sql/utils.py in deco(*a, **kw)
61 def deco(*a, **kw):
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()
~/.local/share/virtualenvs/pydeequ/lib/python3.7/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
--> 328 format(target_id, ".", name), value)
329 else:
330 raise Py4JError(
Py4JJavaError: An error occurred while calling None.com.amazon.deequ.analyzers.Size.
: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
at com.amazon.deequ.analyzers.Size.(Size.scala:37)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
```
**To Reproduce**
Steps to reproduce the behavior:
1. Follow along the tutorial https://github.com/awslabs/python-deequ/blob/master/tutorials/test_data_quality_at_scale.ipynb
2. Attempt to run the following Jupyter cell
```
from pydeequ.analyzers import *
analysisResult = AnalysisRunner(spark) \
.onData(df) \
.addAnalyzer(Size()) \
.addAnalyzer(Completeness("review_id")) \
.addAnalyzer(ApproxCountDistinct("review_id")) \
.addAnalyzer(Mean("star_rating")) \
.addAnalyzer(Compliance("top star_rating", "star_rating >= 4.0")) \
.addAnalyzer(Correlation("total_votes", "star_rating")) \
.addAnalyzer(Correlation("total_votes", "helpful_votes")) \
.run()
analysisResult_df = AnalyzerContext.successMetricsAsDataFrame(spark, analysisResult)
analysisResult_df.show()
```
3. See error in description
**Expected behavior**
Analysis data frame returned per the tutorial
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: Ubuntu 20.04 WSL2
- Browser: Chrome
- Version: 1.0.0
- Python: 3.7.9
- Pyspark: 2.4.0
- Sagemaker-pyspark: 1.4.2
- Py4J: 0.10.7
- Java: OpenJDK 1.8.0_275
Contributor guide
Assessment
This issue has not been assessed yet.