haskell / haskell/haskell-language-server
Flaky testsuit track list
- Dominant language
- Haskell
- Stars
- 3k
- Forks
- 455
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 11
Description
## Cateogries
Fakiness types:
1. busy file handler, should use atomic write instead.
2. progress notification missed, should emit at least start and end for it in lsp during test.
3. vanished pipe, can be fixed by stopping the sender in lsp
## Trackers
Some tests are really falky, it is important for us to take a look and solve them.It is crucial important to the HLS stability.
It should be part of #3736
* [ ] #4270
* [x] #4093 fixed by #4190
* [x] #3333 fixed by #4190
* [x] #4194 fixed by #4190
* [x] error-order-test‘s InternalError over InvalidParams get stuck https://github.com/haskell/haskell-language-server/actions/runs/8868613651/job/24372650102?pr=4195
* [x] ghcide test session-deps-are-picked-up https://github.com/haskell/haskell-language-server/actions/runs/8900448886/job/24442056968?pr=4199
* [x] #4200
* [x] UnitTests's notification handlers run in priority order
```
ghcide/test/exe/UnitTests.hs:91:
expected: [20,19,18,17,16,15,14,13,12,11,10,9,8,7,6,5,4,3,2,1]
but got: [20,19]
```
* [ ] hls-splice:
```
TDeclKindError (golden): FAIL (66.44s)
Timed out waiting to receive a message from the server.
Last message received:
{
"jsonrpc": "2.0",
"method": "$/progress",
"params": {
"token": "35",
"value": {
"kind": "end",
"message": "Finished indexing 3 files"
}
}
}
```
* [ ] ghcide-tests'
``` TemplateHaskell
th-linking-test: FAIL
Exception: Timed out waiting to receive a message from the server.
Last message received:
{
"jsonrpc": "2.0",
"method": "$/progress",
"params": {
"token": "7",
"value": {
"kind": "report",
"message": "5/5"
}
}
}
```
* [x] garbage collection
```
ghcide
garbage collection
dirty keys
are deleted from the state: FAIL (0.15s)
ghcide-test/exe/GarbageCollectionTests.hs:38:
keys were not deleted from the state
```
* [x] https://github.com/haskell/haskell-language-server/issues/4627
PS. I am doing some migration in #4173 collecting these
## Set up long run CI to test the flaky test for 500 times
We can set up a standard to verify a flaky test is gone by running it consectively 500 times.
Since our CI is likely to miss this, we can set up a long running CI elsewhere to track our main branch and run the flaky test we pick, so we can see status of it .
@jhrcek already develop a script to run these test. we can build the long running CI based on it
```
# recommended to build test binary separately and then run it in a loop (to avoid running cabal test in a loop)
# Run tests in a loop
for i in {1..500}; do
echo "Iteration $i" &&
LSP_TEST_LOG_MESSAGES=0 LSP_TEST_LOG_STDERR=0 TASTY_PATTERN="Notification Handlers" $(find dist-newstyle -name ghcide-tests -type f | head -n 1) \
|| {
echo "Warning: error at iteration $i"
break
}; done
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.