cockroachdb / cockroachdb/cockroach
roachtest: auto-kill-threshold should be less sensitive to low test counts
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The `auto-kill-threshold` flag is intended to prevent wasted resources and github noise in test runs where a large percentage of tests have failed due to a common widespread issue that will likely cause subsequent tests to also fail.
We currently run all teamcity runs with `auto-kill-threshold=0.10`. While this value works for a large run, e.g. a nightly where we run hundreds of tests and seeing 30-40 tests fail is abnormal, it commonly cancels engineer invoked runs where there may be less than 10 tests overall. This isn't ideal as the saved resources is minimal and there are no github issues created for manual runs. Currently the only way around this is to remember to set `env.AUTO_KILL_THRESHOLD=1`, which most engineers likely are unaware of.
Instead we could teach the auto kill threshold logic to only kick in for runs with > n total tests. This could be done through an additional `auto-kill-min-tests` flag that we could set to ~50(?).
Alternatively, we could only set `auto-kill-threshold=0.1` on release branches/master, as manually started runs usually never want their runs cancelled. See the `selective_test` logic in `cockroach/build/teamcity/cockroach/nightlies/roachtest_nightly_impl.sh` for a similar approach. This is simpler although the downside is that engineers do sometimes run entire suites where the auto kill threshold would be nice .
Jira issue: CRDB-52139
Contributor guide
Assessment
This issue has not been assessed yet.