awslabs / awslabs/deequ

Add Row Level Semantic Checks and Isolate Failed Checks with RowValidator Object

Open
#281 8 comments 11 reactions 0 assignees View on GitHub
enhancement help-wanted
Dominant language
Scala
Stars
3.6k
Forks
586
Avg merge
13d 13h
Merged PRs (30d)
1

Description

I could not find any way to handle row level checks and isolate non-conforming rows with Deequ's library. Currently, it seems that all the checks are aggregate on the dataset and the RowLevelSchemaValidator handles TypeChecking and structural checks only. This RowValidator can handle semantic checks (i.e. isContainedIn, containsEmail, isInRage, etc.) and identifies which checks failed so that bad records can be put into a reject queue for some data steward to examine. More on this issue can be found here https://github.com/awslabs/deequ/issues/223.

Some example code to show how to use this tool:
`
val dataset = spark.read.format("csv").option("header","true").load("path_to_data.csv")

val checks = RowLevelChecks()
.containsEmail("email")
.containsCreditCardNumber("creditcard")
.containsSocialSecurityNumber("ssn")
.containsURL("url")
.isNonNegative("nonnegative")
.isInRange("nonnegative", -5, 200)
.isLessThanOrEqualTo("num1", "num2")
.isLessThanOrEqualTo("num1", 50)
.isContainedIn("country", Array("USA", "MEX", "CAN"))

val result = RowValidator.validate(dataset, checks)

val validRows = result.validRows // dataframe with rows passing all checks
val numValidRows = result.numValidRows

val invalidRows = result.invalidRows // dataframe with non-conforming rows and column listing checks that failed
val numInvalidRows = result.numInValidRows
`

Contributor guide

Open the contributing guide

Research direction

Start with the RowLevelChecks and RowValidator entry points shown in the example, and compare them with the existing RowLevelSchemaValidator behavior. Define how semantic checks such as isContainedIn, containsEmail, and isInRange report failures per row. Done means validation returns validRows and invalidRows, counts for each, and the failed checks associated with invalid records.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.