[Feature] Support unit testing for unordered arrays
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this a new bug in dbt-core?
- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug
### Current Behavior
Using Apache Spark as a back end, when my ETL creates an array the order of that array is not deterministic. In order for my unit tests to work with arrays of length 2 or greater I will need to sort the arrays in my model which will cost additional processing on my production job.
### Expected Behavior
Ideally there would be some way to implement a manual ETL step between my result and comparing expected to enable cases like this to all be handled regardless.
As a minimum though, it would be helpful for the unit tests yaml to have a flag I can set to ask it to sort arrays and which defaults to False/no sort but this seems like other edge cases would crop up.
### Steps To Reproduce
0. Have DBT setup with Apache spark via SQL.
1. Make a unit test for a model that creates an array in the model output.
2. Make sure the setup produces arrays with length 2 or greater.
3. Have several arrays created at once each time you run the tests.
4. If you have enough arrays in the results it will fail in a different way every time you run it.
### Relevant log output
```shell
Values collected into arrays from rows a, b in the source table, result:
actual differs from expected:
@@,col1,col2,col3,...
→,['col1_rowb' 'col1_rowa']→['col1_rowa' 'col1_rowb'],['col2_rowa' 'col2_rowb'],['col3_rowb' 'col3_rowa']→['col3_rowa' 'col3_rowb'],...
```
### Environment
```markdown
- OS: Linux
- Python: 3.10.14
- dbt: 1.8.3
```
### Which database adapter are you using with dbt?
other (mention it in "Additional Context")
### Additional Context
Databricks adapter.
Contributor guide
Assessment
This issue has not been assessed yet.