awslabs / awslabs/deequ

RowLevelSchema().withIntColumn is reporting non integers in validRows

Open
#262 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Scala
Stars
3.6k
Forks
586
Avg merge
13d 13h
Merged PRs (30d)
1

Description

I changed line 124 in RowLevelSchemaValidatorTest to be non integer. Code is not reporting that value in invalid results.
Integer value is reported in validRows. I changer 123 to 1.23 and got 1 in validRows. Below is the code from RowLevelSchemaValidatorTest. Only difference is that 123 is now 1.23

val data = Seq(
**"1.23", // original val was "123",**
"N/A",
"456",
"999999",
"-9",
"-100000",
null
).toDF("id")

val schema = RowLevelSchema()
.withIntColumn("id", isNullable = false, minValue = Some(-10), maxValue = Some(1000))

val result = RowLevelSchemaValidator.validate(data, schema)

assert(result.numValidRows == 3)
val validIds = result.validRows.select("id").collect.map { _.getInt(0) }.toSet
assert(validIds.size == result.numValidRows)
assert(validIds.contains(1)) // original code assert(validIds.contains(123))
assert(validIds.contains(456))
assert(validIds.contains(-9))

Contributor guide

Open the contributing guide

Research direction

Start with RowLevelSchemaValidatorTest around line 124 and reproduce the case using RowLevelSchema().withIntColumn("id") and RowLevelSchemaValidator.validate. Trace how the validator handles the "1.23" value and how validRows and invalid results are assembled. Done means non-integer input is not reported as a valid integer row and the test assertions reflect the correct result.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.