awslabs / awslabs/python-deequ

Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!

Open
#22 3 comments 0 reactions 2 assignees Claimed by @gucciwang View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
826
Forks
158
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue:**
While running the tutorial code on AWS Glue, I can't get the checks to pass when there is an assertion. It seems to be a type issue.

"Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!"

**Environment**:
Glue Development Endpoint
Spark Version: 2.4
Python Version: 3
Pydeequ version: 0.1.5
Deequ jar: 1.0.3

**Code**:
Using the tutorial sample code:

df = spark.sparkContext.parallelize([
Row(a="foo", b=1, c=5),
Row(a="bar", b=2, c=6),
Row(a="baz", b=3, c=None)]).toDF()

check = Check(spark, CheckLevel.Warning, "Review Check")

checkResult = VerificationSuite(spark) \
.onData(df) \
.addCheck(
check.hasSize(lambda x: x >= 3) \
.hasMin("b", lambda x: x == 0) \
.isComplete("c") \
.isUnique("a") \
.isContainedIn("a", ["foo", "bar", "baz"]) \
.isNonNegative("b")) \
.run()

checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, checkResult)
checkResult_df.show()

_Output_:
+------------+-----------+------------+--------------------+-----------------+--------------------+
| check|check_level|check_status| constraint|constraint_status| constraint_message|
+------------+-----------+------------+--------------------+-----------------+--------------------+
|Review Check| Warning| Warning|SizeConstraint(Si...| Failure|Can't execute the...|
|Review Check| Warning| Warning|MinimumConstraint...| Failure|Can't execute the...|
|Review Check| Warning| Warning|CompletenessConst...| Failure|Value: 0.66666666...|
|Review Check| Warning| Warning|UniquenessConstra...| Success| |
|Review Check| Warning| Warning|ComplianceConstra...| Success| |
|Review Check| Warning| Warning|ComplianceConstra...| Success| |
+------------+-----------+------------+--------------------+-----------------+--------------------+

_Expanding the two first rows with failure_:

Row(check='Review Check', check_level='Warning', check_status='Warning', constraint='SizeConstraint(Size(None))', constraint_status='Failure', constraint_message="**Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!**")
Row(check='Review Check', check_level='Warning', check_status='Warning', constraint='MinimumConstraint(Minimum(b,None))', constraint_status='Failure', constraint_message="**Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!**")

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.