`batches_to_sort_string` differing from similar implementation in `assert_batches_sorted_eq`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
I was migrating tests to `insta` in a PR #15248 and got a problem. For same expected output in a test, I was finding differing old and new snapshots while using `batches_to_sort_string` and no differing snapshots while using `assert_batches_sorted_eq`. I did not encounter this issue while migrating many other tests in `/datafusion/physical-plan`, this weirdly was the first time I encountered this issue.
Edit: Also referencing at the following PR comment https://github.com/apache/datafusion/pull/15288#discussion_r2004177854 . This is also a problem discovered so far.
Previous code (using assert_batches_sorted_eq) :
```
let expected = [
"+---+---+---+----+---+---+",
"| a | b | c | a | b | c |",
"+---+---+---+----+---+---+",
"| | | | 30 | 3 | 6 |",
"| | | | 40 | 4 | 4 |",
"| 2 | 7 | 9 | 10 | 2 | 7 |",
"| 2 | 7 | 9 | 20 | 2 | 5 |",
"| 0 | 4 | 7 | | | |",
"| 1 | 5 | 8 | | | |",
"| 2 | 8 | 1 | | | |",
"+---+---+---+----+---+---+",
];
assert_batches_sorted_eq!(expected, &batches);
```
New code (using batches_to_sort_string) :
```
allow_duplicates! {
assert_snapshot!(batches_to_sort_string(&batches), @r#"
+---+---+---+----+---+---+
| a | b | c | a | b | c |
+---+---+---+----+---+---+
| | | | 30 | 3 | 6 |
| | | | 40 | 4 | 4 |
| 2 | 7 | 9 | 10 | 2 | 7 |
| 2 | 7 | 9 | 20 | 2 | 5 |
| 0 | 4 | 7 | | | |
| 1 | 5 | 8 | | | |
| 2 | 8 | 1 | | | |
+---+---+---+----+---+---+
"#)
}
```
In both cases, I had made sure several times that the expected output is the same.
I am getting the following output while using new code:

### To Reproduce
In `/datafusion/physical-plan/src/joins/hash_join.rs`,
replace following part in `async fn join_full_with_filter(batch_size: usize) -> Result<()>`:
```
let expected = [
"+---+---+---+----+---+---+",
"| a | b | c | a | b | c |",
"+---+---+---+----+---+---+",
"| | | | 30 | 3 | 6 |",
"| | | | 40 | 4 | 4 |",
"| 2 | 7 | 9 | 10 | 2 | 7 |",
"| 2 | 7 | 9 | 20 | 2 | 5 |",
"| 0 | 4 | 7 | | | |",
"| 1 | 5 | 8 | | | |",
"| 2 | 8 | 1 | | | |",
"+---+---+---+----+---+---+",
];
assert_batches_sorted_eq!(expected, &batches);
```
with
```
allow_duplicates! {
assert_snapshot!(batches_to_sort_string(&batches), @r#"
+---+---+---+----+---+---+
| a | b | c | a | b | c |
+---+---+---+----+---+---+
| | | | 30 | 3 | 6 |
| | | | 40 | 4 | 4 |
| 2 | 7 | 9 | 10 | 2 | 7 |
| 2 | 7 | 9 | 20 | 2 | 5 |
| 0 | 4 | 7 | | | |
| 1 | 5 | 8 | | | |
| 2 | 8 | 1 | | | |
+---+---+---+----+---+---+
"#)
}
```
### Expected behavior
Similar results for both the tests.
### Additional context
I did not encounter this issue while migrating many other tests in `/datafusion/physical-plan`, this weirdly was the first time I encountered this issue.
Contributor guide
Research direction
Reproduce the issue in datafusion/physical-plan/src/joins/hash_join.rs, in join_full_with_filter, using the two test forms shown. Read the implementations of batches_to_sort_string and assert_batches_sorted_eq and compare how each orders the batches, including duplicates. Done means both assertions produce equivalent results for this case and the relevant test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100