awslabs / awslabs/python-deequ
Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the issue:**
While running the tutorial code on AWS Glue, I can't get the checks to pass when there is an assertion. It seems to be a type issue.
"Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!"
**Environment**:
Glue Development Endpoint
Spark Version: 2.4
Python Version: 3
Pydeequ version: 0.1.5
Deequ jar: 1.0.3
**Code**:
Using the tutorial sample code:
df = spark.sparkContext.parallelize([
Row(a="foo", b=1, c=5),
Row(a="bar", b=2, c=6),
Row(a="baz", b=3, c=None)]).toDF()
check = Check(spark, CheckLevel.Warning, "Review Check")
checkResult = VerificationSuite(spark) \
.onData(df) \
.addCheck(
check.hasSize(lambda x: x >= 3) \
.hasMin("b", lambda x: x == 0) \
.isComplete("c") \
.isUnique("a") \
.isContainedIn("a", ["foo", "bar", "baz"]) \
.isNonNegative("b")) \
.run()
checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, checkResult)
checkResult_df.show()
_Output_:
+------------+-----------+------------+--------------------+-----------------+--------------------+
| check|check_level|check_status| constraint|constraint_status| constraint_message|
+------------+-----------+------------+--------------------+-----------------+--------------------+
|Review Check| Warning| Warning|SizeConstraint(Si...| Failure|Can't execute the...|
|Review Check| Warning| Warning|MinimumConstraint...| Failure|Can't execute the...|
|Review Check| Warning| Warning|CompletenessConst...| Failure|Value: 0.66666666...|
|Review Check| Warning| Warning|UniquenessConstra...| Success| |
|Review Check| Warning| Warning|ComplianceConstra...| Success| |
|Review Check| Warning| Warning|ComplianceConstra...| Success| |
+------------+-----------+------------+--------------------+-----------------+--------------------+
_Expanding the two first rows with failure_:
Row(check='Review Check', check_level='Warning', check_status='Warning', constraint='SizeConstraint(Size(None))', constraint_status='Failure', constraint_message="**Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!**")
Row(check='Review Check', check_level='Warning', check_status='Warning', constraint='MinimumConstraint(Minimum(b,None))', constraint_status='Failure', constraint_message="**Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!**")
Contributor guide
Assessment
This issue has not been assessed yet.