Improve BATS test execution for Mac and Windows
- Dominant language
- Go
- Stars
- 24.4k
- Forks
- 873
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 108
Description
We don't currently run the Dolt BATS test suite on Mac and Windows platforms in a way that ensures regressions for those platforms don't get merged to main. This means platform-specific regressions that our BATS tests _could_ catch can sometimes get released. We saw an example of that today with a windows-specific `drop database` issue (https://github.com/dolthub/dolt/issues/7492).
For Unix/Ubuntu, the BATS test suite currently runs on every PR going to the main branch of the dolt repo, and runs in parallel on AWS Lambda, which gives us good coverage of Unix/Ubuntu environments and only takes around 5 or 6 minutes to execute.
For MacOS, the BATS test suite takes 90+ minutes to run on every PR and often had contention with Mac runners which added further latency. We [moved Mac BATS tests to run nightly](https://github.com/dolthub/dolt/pull/7495), which will still alert us to failures, but won't prevent those regressions from reaching main and being released.
For Windows, the BATS test suite hasn't been running anywhere for over a year. Due to not being used, these tests have decayed and need work to get them passing again on Windows. Many individual tests also explicitly skip windows, which reduces the coverage of the test suite.
Ideally, we want to prevent platform-specific regressions from being released to customers, while also balancing the need for developers to run tests in PRs in a reasonable amount of time. The Lambda parallelization has worked extremely well for the Unix/Ubuntu tests, but is not as straightforward for Mac and Windows, so another approach my be necessary.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.