dbt-labs / dbt-labs/dbt

[Feature] Support unit testing for unordered arrays

Open
#10,573 2 comments 0 reactions 0 assignees View on GitHub
engine:v1 status:triage type:feature unit tests
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

Using Apache Spark as a back end, when my ETL creates an array the order of that array is not deterministic. In order for my unit tests to work with arrays of length 2 or greater I will need to sort the arrays in my model which will cost additional processing on my production job.

### Expected Behavior

Ideally there would be some way to implement a manual ETL step between my result and comparing expected to enable cases like this to all be handled regardless.

As a minimum though, it would be helpful for the unit tests yaml to have a flag I can set to ask it to sort arrays and which defaults to False/no sort but this seems like other edge cases would crop up.

### Steps To Reproduce

0. Have DBT setup with Apache spark via SQL.
1. Make a unit test for a model that creates an array in the model output.
2. Make sure the setup produces arrays with length 2 or greater.
3. Have several arrays created at once each time you run the tests.
4. If you have enough arrays in the results it will fail in a different way every time you run it.

### Relevant log output

```shell
Values collected into arrays from rows a, b in the source table, result:

actual differs from expected:
@@,col1,col2,col3,...
→,['col1_rowb' 'col1_rowa']→['col1_rowa' 'col1_rowb'],['col2_rowa' 'col2_rowb'],['col3_rowb' 'col3_rowa']→['col3_rowa' 'col3_rowb'],...
```

### Environment

```markdown
- OS: Linux
- Python: 3.10.14
- dbt: 1.8.3
```

### Which database adapter are you using with dbt?

other (mention it in "Additional Context")

### Additional Context

Databricks adapter.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.