dolthub / dolthub/dolt

Improve BATS test execution for Mac and Windows

Open
#7,498 0 comments 0 reactions 0 assignees View on GitHub
testing
Dominant language
Go
Stars
24.4k
Forks
873
Avg merge
1d 5h
Merged PRs (30d)
108

Description

We don't currently run the Dolt BATS test suite on Mac and Windows platforms in a way that ensures regressions for those platforms don't get merged to main. This means platform-specific regressions that our BATS tests _could_ catch can sometimes get released. We saw an example of that today with a windows-specific `drop database` issue (https://github.com/dolthub/dolt/issues/7492).

For Unix/Ubuntu, the BATS test suite currently runs on every PR going to the main branch of the dolt repo, and runs in parallel on AWS Lambda, which gives us good coverage of Unix/Ubuntu environments and only takes around 5 or 6 minutes to execute.

For MacOS, the BATS test suite takes 90+ minutes to run on every PR and often had contention with Mac runners which added further latency. We [moved Mac BATS tests to run nightly](https://github.com/dolthub/dolt/pull/7495), which will still alert us to failures, but won't prevent those regressions from reaching main and being released.

For Windows, the BATS test suite hasn't been running anywhere for over a year. Due to not being used, these tests have decayed and need work to get them passing again on Windows. Many individual tests also explicitly skip windows, which reduces the coverage of the test suite.

Ideally, we want to prevent platform-specific regressions from being released to customers, while also balancing the need for developers to run tests in PRs in a reasonable amount of time. The Lambda parallelization has worked extremely well for the Unix/Ubuntu tests, but is not as straightforward for Mac and Windows, so another approach my be necessary.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.