haskell / haskell/haskell-language-server

CI: Dont block CI on flaky tests

Open
#3,581 1 comment 1 reaction 0 assignees View on GitHub
CI type: enhancement
Dominant language
Haskell
Stars
3k
Forks
455
Avg merge
3d 21h
Merged PRs (30d)
13

Description

## Is your enhancement request related to a problem? Please describe.

CI is currently getting stuck a lot. This is terrible for contributors, even simple, small changes can take weeks to merge because CI is taking multiple runs to finally succeed. The main reason for these CI issues are flaky tests, i.e. tests that sometimes fail and sometimes succeed. Contributors have no way of fixing these failing tests, waiting for a contributor who has the permission to restart failing CI jobs.

I think this drags down morale and exhausts possible contributors.

## Describe the solution you'd like

As long as we cannot avoid having flaky tests, these flaky tests should not block CI.
Thus, I propose to run the tests in CI, but no longer fail a CI job if a flaky test fails.
We introduce a new combinator

```haskell
-- | Mark test as flaky for a specific target
knownFlakyFor :: BrokenTarget -> String -> TestTree -> TestTree
knownFlakyFor ...

-- | Mark test as flaky for all GHC versions
knownFlaky :: String -> TestTree -> TestTree
knownFlaky ...
```

which works like the existing `knownBrokenFor` and `ignoreFor` functions and ties into the same infrastructure, by allowing us to specify on which platform the tests are flaky. Note, however, it stands to reason, tests are usually not flaky because of GHC but LSP things and parallelism.

I want to mark all tests that are known flaky with these combinators, together with possible reasons.

~~CI should no longer retry a single test suite, but whenever a test is demonstrable flaky, then it should be marked as such. It will also reduce the repetitive code in test.yml where we just blindly retry all test-suites.~~

Unfortunately, in experiments, it has shown that basically all lsp-test tests are prone to flakiness. However, they are so flaky, that re-running the test suite usually works fine to make the test succeed. Thus, we only mark tests as flaky that are too flaky, i.e., if they fail twice in a row. With that approach, I hope we can get CI to succeed reliably again.

## Describe alternatives you've considered

Fix all the flaky tests, by either improving lsp-test to be easier to use correctly, migrating integration-tests to unit-tests or fixing the test-cases by introducing some kind of sync points.

I also don't think we run the risk of marking a test as flaky and forgetting about them instead of fixing them. Currently, if a test is flaky, we just restart the CI, usually not even looking at what test failed in particular (correct me if I am wrong).
Testsuite maintenance and bitrot is an important topic anyway, and we need to review the tests that are marked as broken as well, so we just document the status quo in the code, in my opinion.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing knownBrokenFor and ignoreFor combinators and the CI logic in test.yml. Review the lsp-test tests that are described as flaky and determine how knownFlakyFor and knownFlaky should connect to the existing infrastructure. Done means repeatedly flaky tests are marked with their applicable platform and possible reason without blocking CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.