haskell / haskell/haskell-language-server

Flaky testsuit track list

Open
#4,193 1 comment 0 reactions 0 assignees View on GitHub
CI component: cli flaky test type: bug type: enhancement
Dominant language
Haskell
Stars
3k
Forks
455
Avg merge
2d 19h
Merged PRs (30d)
11

Description

## Cateogries
Fakiness types:
1. busy file handler, should use atomic write instead.
2. progress notification missed, should emit at least start and end for it in lsp during test.
3. vanished pipe, can be fixed by stopping the sender in lsp

## Trackers
Some tests are really falky, it is important for us to take a look and solve them.It is crucial important to the HLS stability.
It should be part of #3736
* [ ] #4270
* [x] #4093 fixed by #4190
* [x] #3333 fixed by #4190
* [x] #4194 fixed by #4190
* [x] error-order-test‘s InternalError over InvalidParams get stuck https://github.com/haskell/haskell-language-server/actions/runs/8868613651/job/24372650102?pr=4195
* [x] ghcide test session-deps-are-picked-up https://github.com/haskell/haskell-language-server/actions/runs/8900448886/job/24442056968?pr=4199
* [x] #4200
* [x] UnitTests's notification handlers run in priority order
```
ghcide/test/exe/UnitTests.hs:91:
expected: [20,19,18,17,16,15,14,13,12,11,10,9,8,7,6,5,4,3,2,1]
but got: [20,19]
```
* [ ] hls-splice:
```
TDeclKindError (golden): FAIL (66.44s)
Timed out waiting to receive a message from the server.
Last message received:
{
"jsonrpc": "2.0",
"method": "$/progress",
"params": {
"token": "35",
"value": {
"kind": "end",
"message": "Finished indexing 3 files"
}
}
}
```
* [ ] ghcide-tests'
``` TemplateHaskell
th-linking-test: FAIL
Exception: Timed out waiting to receive a message from the server.
Last message received:
{
"jsonrpc": "2.0",
"method": "$/progress",
"params": {
"token": "7",
"value": {
"kind": "report",
"message": "5/5"
}
}
}
```
* [x] garbage collection
```
ghcide
garbage collection
dirty keys
are deleted from the state: FAIL (0.15s)
ghcide-test/exe/GarbageCollectionTests.hs:38:
keys were not deleted from the state
```
* [x] https://github.com/haskell/haskell-language-server/issues/4627

PS. I am doing some migration in #4173 collecting these

## Set up long run CI to test the flaky test for 500 times
We can set up a standard to verify a flaky test is gone by running it consectively 500 times.
Since our CI is likely to miss this, we can set up a long running CI elsewhere to track our main branch and run the flaky test we pick, so we can see status of it .

@jhrcek already develop a script to run these test. we can build the long running CI based on it
```
# recommended to build test binary separately and then run it in a loop (to avoid running cabal test in a loop)
# Run tests in a loop
for i in {1..500}; do
echo "Iteration $i" &&
LSP_TEST_LOG_MESSAGES=0 LSP_TEST_LOG_STDERR=0 TASTY_PATTERN="Notification Handlers" $(find dist-newstyle -name ghcide-tests -type f | head -n 1) \
|| {
echo "Warning: error at iteration $i"
break
}; done
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.