Exception in User Class: java.lang.StackOverflowError in deequ
- Dominant language
- Scala
- Stars
- 3.6k
- Forks
- 586
- Avg merge
- 13d 13h
- Merged PRs (30d)
- 1
Description
I am running deequ and found following bug.
When we have deequ.checks under 100 for one dataframe or table deequ working without issue .
```
import com.amazon.deequ.VerificationResult.checkResultsAsDataFrame
import com.amazon.deequ.checks.{Check, CheckLevel}
import com.amazon.deequ.constraints.ConstrainableDataTypes
val _check = Check(CheckLevel.Error, "Data Validation Check")
.isComplete("First_Name")
.isComplete("Month_Name_of_Joining")
.isContainedIn("Month_Name_of_Joining", Array("August", "July", "January", "April", "December", "November", "February", "March", "June", "September", "May", "October"))
.isComplete("Phone_No")
.containsEmail("E_Mail")
.containsSocialSecurityNumber("SSN")
.isContainedIn("Day_of_Joining", 1, 31, includeLowerBound = true, includeUpperBound = true)
.hasPattern("Phone_No", """^[+]*[(]{0,1}[0-9]{1,4}[)]{0,1}[-\s\./0-9]*$""".r)
.hasDataType("Emp_ID", ConstrainableDataTypes.Integral)
.isUnique("Emp_ID")
val verificationResult: VerificationResult = { VerificationSuite()
.onData()
.addCheck(_check)
.run()
}
val resultDataFrame = checkResultsAsDataFrame(spark, verificationResult)
display (resultDataFrame)
```
Once check size more then 200 column deequ VerificationResult.checkResultsAsDataFrame start throwing java.lang.StackOverflowError .
You need to add really log checks in deequ like 250 or 300 and run for 100 rows dataframe defeat will reproduce .
Contributor guide
Research direction
Start at VerificationResult.checkResultsAsDataFrame and reproduce the failure with 250–300 checks on a 100-row Spark DataFrame. Trace the StackOverflowError from that entry point and verify that the result DataFrame can be produced without overflowing when handling the larger check set.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100