awslabs / awslabs/deequ

Exception in User Class: java.lang.StackOverflowError in deequ

Open
#409 1 comment 0 reactions 0 assignees View on GitHub
bug spark-compatibility
Dominant language
Scala
Stars
3.6k
Forks
586
Avg merge
13d 13h
Merged PRs (30d)
1

Description

I am running deequ and found following bug.

When we have deequ.checks under 100 for one dataframe or table deequ working without issue .

```
import com.amazon.deequ.VerificationResult.checkResultsAsDataFrame
import com.amazon.deequ.checks.{Check, CheckLevel}
import com.amazon.deequ.constraints.ConstrainableDataTypes

val _check = Check(CheckLevel.Error, "Data Validation Check")
.isComplete("First_Name")
.isComplete("Month_Name_of_Joining")
.isContainedIn("Month_Name_of_Joining", Array("August", "July", "January", "April", "December", "November", "February", "March", "June", "September", "May", "October"))
.isComplete("Phone_No")
.containsEmail("E_Mail")
.containsSocialSecurityNumber("SSN")
.isContainedIn("Day_of_Joining", 1, 31, includeLowerBound = true, includeUpperBound = true)
.hasPattern("Phone_No", """^[+]*[(]{0,1}[0-9]{1,4}[)]{0,1}[-\s\./0-9]*$""".r)
.hasDataType("Emp_ID", ConstrainableDataTypes.Integral)
.isUnique("Emp_ID")


val verificationResult: VerificationResult = { VerificationSuite()
.onData()
.addCheck(_check)
.run()
}
val resultDataFrame = checkResultsAsDataFrame(spark, verificationResult)
display (resultDataFrame)

```

Once check size more then 200 column deequ VerificationResult.checkResultsAsDataFrame start throwing java.lang.StackOverflowError .

You need to add really log checks in deequ like 250 or 300 and run for 100 rows dataframe defeat will reproduce .

Contributor guide

Open the contributing guide

Research direction

Start at VerificationResult.checkResultsAsDataFrame and reproduce the failure with 250–300 checks on a 100-row Spark DataFrame. Trace the StackOverflowError from that entry point and verify that the result DataFrame can be produced without overflowing when handling the larger check set.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.