haskell / haskell/haskell-language-server

Overhauling the HLS testsuite

Open
#3,736 9 comments 12 reactions 0 assignees View on GitHub
type: enhancement
Dominant language
Haskell
Stars
3k
Forks
455
Avg merge
4d 3h
Merged PRs (30d)
12

Description

## Problem

The test suite of HLS is gigantic. That's great!
Running the test suite of HLS takes a long time. That's bad.
Even worse, the test suite has become immensely flaky. Usually, we require at least one CI rerun for failed CI jobs per commit. Then we add insult to injury, since tests fail for components that the PR doesn't even touch.
We concluded, that a lot of the flakiness and bad performance originates from lsp-test which is basically a parser generator for parsing a very specific order of lsp messages.

However, HLS is not the most deterministic language server out there, so sometimes messages come out of the expected order. For example, published diagnostics for module A are sent before module B, which leads the test to fail. Rerunning the same test might yield a different order of messages, causing the test to succeed.

The performance and flakiness of our CI has become a real problem, which is why need to come up with solutions.

As a comparison, `rust-analyzers` while test-suite takes 1.30-min while HLS usually takes up to an hour. It is our dedicated goal to be within the 5-min mark for running our *entire* test suite.

## Solution

There are many ways to reach our destination. So we outline a couple here:

1) Reducing flakiness in `lsp-test`. We have already identified some common sources of flakiness. The most important is loading too many modules into the same test session. Making sure, each test-case loads only exactly what it needs for the test, reduces the flakiness.
2) I argue that `lsp-test` is often the wrong tool for testing. To be precise, LSP is the wrong API boundary for most of our tests! `lsp-test` sets up an entire HLS session, negotiates capabilities, etc... All not needed for testing a plugin Code Action! Thus, we take https://matklad.github.io/2021/05/31/how-to-test.html as our role model and try to gain as much as possible from it.

The rest of the issue, will outline how we envision fixing the test suite using the knowledge gained from point 2. We heavily piggyback from the various pieces of wisdom from https://matklad.github.io/2021/05/31/how-to-test.html.

We have some goals for our test suite:
* Avoid IO as much as possible (it is slow, use the VFS if possible)
* Tests should be as stable as possible
* Changing internals of a plugin should not affect the test suite. We need to find a fitting API boundary to test that is relatively stable.
* A single test should test one thing
* Tests should be quick and stable
* Quick feedback is essential for good developer experience
* Slow tests are opt-in
* We cannot and do not want to avoid `lsp-test` where appropriate. There are many examples of tests that absolutely should be using `lsp-test`. However, these tests usually don't need to be run outside of CI.

Now, we discuss some potential API boundaries and test API for **Plugins**. Ghcide internals may require a different API boundary.

### Plugin Tests

Plugins have a rather well-defined interface that is declaratively tested by lsp-test right now. Our tests should be as declarative as possible.

* Test data set up should be defined declaratively
* The ide state (typechecked, parsing, desugar) should be defined declaratively
* Isolated
* Testdata should be moved to a temp directory
* I am a bit skeptical this is a real improvement. I hoped, this will allow us to run more tests in parallel.

For the API boundaries that we want to test, we identify the following:

* Handlers
* Handlers naturally lend themselves as an API boundary.
* Commands
* Slightly awkward interface, since commands can only send server requests to change to workspace files, they are still a natural API boundary
* Rules
* While very much internal implementation detail, they seem interesting enough to accept the violation of the principles mentioned above.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.