holdenk / holdenk/spark-testing-base
assertDataFrameEquals message says that 2 schemas are different after printing out the exact same schemas
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
assertion that failed:
```
assertDataFrameEquals(
List(
(Some(1L), "whiskey"),
(Some(2L), "bianca")
).toDF("id", "name").orderBy("id"),
spark.table("test_animals.dogs").orderBy("id")
)
```
as you can see, the message below shows the same schemas after saying they aren't equal
```
[info] - incremental append *** FAILED ***
[info] StructType(StructField(id,LongType,true), StructField(name,StringType,true)) did not equal StructType(StructField(id,LongType,true), StructField(name,StringType,true)) (TestSuite.scala:13)
```
This might be related to https://github.com/holdenk/spark-testing-base/issues/179 but I don't know exactly what metadata is different or if there's a clean workaround.
Contributor guide
Research direction
Start at assertDataFrameEquals and the failure reported at TestSuite.scala:13. Compare the schema values and metadata used by the assertion, using the linked issue for context about possible differences. Done means the assertion explains the actual difference instead of reporting identical-looking schemas as unequal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100