holdenk / holdenk/spark-testing-base

assertDataFrameEquals message says that 2 schemas are different after printing out the exact same schemas

Open
#236 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

assertion that failed:

```
assertDataFrameEquals(
List(
(Some(1L), "whiskey"),
(Some(2L), "bianca")
).toDF("id", "name").orderBy("id"),
spark.table("test_animals.dogs").orderBy("id")
)
```

as you can see, the message below shows the same schemas after saying they aren't equal

```
[info] - incremental append *** FAILED ***
[info] StructType(StructField(id,LongType,true), StructField(name,StringType,true)) did not equal StructType(StructField(id,LongType,true), StructField(name,StringType,true)) (TestSuite.scala:13)
```

This might be related to https://github.com/holdenk/spark-testing-base/issues/179 but I don't know exactly what metadata is different or if there's a clean workaround.

Contributor guide

Open the contributing guide

Research direction

Start at assertDataFrameEquals and the failure reported at TestSuite.scala:13. Compare the schema values and metadata used by the assertion, using the linked issue for context about possible differences. Done means the assertion explains the actual difference instead of reporting identical-looking schemas as unequal.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.