Add Row Level Semantic Checks and Isolate Failed Checks with RowValidator Object
- Dominant language
- Scala
- Stars
- 3.6k
- Forks
- 586
- Avg merge
- 13d 13h
- Merged PRs (30d)
- 1
Description
I could not find any way to handle row level checks and isolate non-conforming rows with Deequ's library. Currently, it seems that all the checks are aggregate on the dataset and the RowLevelSchemaValidator handles TypeChecking and structural checks only. This RowValidator can handle semantic checks (i.e. isContainedIn, containsEmail, isInRage, etc.) and identifies which checks failed so that bad records can be put into a reject queue for some data steward to examine. More on this issue can be found here https://github.com/awslabs/deequ/issues/223.
Some example code to show how to use this tool:
`
val dataset = spark.read.format("csv").option("header","true").load("path_to_data.csv")
val checks = RowLevelChecks()
.containsEmail("email")
.containsCreditCardNumber("creditcard")
.containsSocialSecurityNumber("ssn")
.containsURL("url")
.isNonNegative("nonnegative")
.isInRange("nonnegative", -5, 200)
.isLessThanOrEqualTo("num1", "num2")
.isLessThanOrEqualTo("num1", 50)
.isContainedIn("country", Array("USA", "MEX", "CAN"))
val result = RowValidator.validate(dataset, checks)
val validRows = result.validRows // dataframe with rows passing all checks
val numValidRows = result.numValidRows
val invalidRows = result.invalidRows // dataframe with non-conforming rows and column listing checks that failed
val numInvalidRows = result.numInValidRows
`
Contributor guide
Research direction
Start with the RowLevelChecks and RowValidator entry points shown in the example, and compare them with the existing RowLevelSchemaValidator behavior. Define how semantic checks such as isContainedIn, containsEmail, and isInRange report failures per row. Done means validation returns validRows and invalidRows, counts for each, and the failed checks associated with invalid records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100