RowLevelSchema().withIntColumn is reporting non integers in validRows
- Dominant language
- Scala
- Stars
- 3.6k
- Forks
- 586
- Avg merge
- 13d 13h
- Merged PRs (30d)
- 1
Description
I changed line 124 in RowLevelSchemaValidatorTest to be non integer. Code is not reporting that value in invalid results.
Integer value is reported in validRows. I changer 123 to 1.23 and got 1 in validRows. Below is the code from RowLevelSchemaValidatorTest. Only difference is that 123 is now 1.23
val data = Seq(
**"1.23", // original val was "123",**
"N/A",
"456",
"999999",
"-9",
"-100000",
null
).toDF("id")
val schema = RowLevelSchema()
.withIntColumn("id", isNullable = false, minValue = Some(-10), maxValue = Some(1000))
val result = RowLevelSchemaValidator.validate(data, schema)
assert(result.numValidRows == 3)
val validIds = result.validRows.select("id").collect.map { _.getInt(0) }.toSet
assert(validIds.size == result.numValidRows)
assert(validIds.contains(1)) // original code assert(validIds.contains(123))
assert(validIds.contains(456))
assert(validIds.contains(-9))
Contributor guide
Research direction
Start with RowLevelSchemaValidatorTest around line 124 and reproduce the case using RowLevelSchema().withIntColumn("id") and RowLevelSchemaValidator.validate. Trace how the validator handles the "1.23" value and how validRows and invalid results are assembled. Done means non-integer input is not reported as a valid integer row and the test assertions reflect the correct result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100