haskell / haskell/hackage-server

Slow HTTP responses when the cache is updating

Open
#1,138 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
467
Forks
225
PR merge metrics
No merged PRs in 30d

Description

I have an internal hackage-server running, which is deployed with a http based health checker that does a GET request to `/` to see whether it's up.

The hackage-server was periodically getting marked as down and looking at the logs it seems this is because it's taking too long to respond while it's doing the cache updates. We could start to use a health checker that doesn't use http, but really it shouldn't be taking this long to respond in the first place. Logs look like this:
```
127.0.0.1 - [10/Oct/2022:14:18:54 +0000] "GET /00-index.tar.gz 1.1" 301 -1 "" "cabal-install/3.2.0.0 (linux; x86_64)"
127.0.0.1 - [10/Oct/2022:14:18:54 +0000] "GET /packages/index.tar.gz 1.1" 304 -1 "" "cabal-install/3.2.0.0 (linux; x86_64)"
127.0.0.1 - [10/Oct/2022:14:18:54 +0000] "GET /packages/docs.json 1.1" 200 -1 "" "hackage-build/0.5.1"
127.0.0.1 - [10/Oct/2022:14:18:55 +0000] "GET /packages/candidates/docs.json 1.1" 200 -1 "" "hackage-build/0.5.1"
127.0.0.1 - [10/Oct/2022:14:19:01 +0000] "GET / 1.0" 200 -1 "" "bdns_healthchecker/2022.922.234"
Cache 'packages index page (by category)' updated (item update). time: 12.955302227s
Cache 'packages index page (by name)' updated (item update). time: 0.81750265s
Cache 'browse packages' updated (item update). time: 12.250529765s
127.0.0.1 - [10/Oct/2022:14:19:33 +0000] "GET / 1.0" 200 -1 "" "healthchecker"
HTTP request failed with: Network.Socket.sendBuf: resource vanished (Broken pipe)
127.0.0.1 - [10/Oct/2022:14:19:33 +0000] "GET / 1.0" 200 -1 "" "healthchecker"
127.0.0.1 - [10/Oct/2022:14:19:33 +0000] "GET / 1.0" 200 -1 "" "healthchecker"
HTTP request failed with: Network.Socket.sendBuf: resource vanished (Broken pipe)
127.0.0.1 - [10/Oct/2022:14:19:41 +0000] "GET / 1.0" 200 -1 "" "healthchecker"
```

This seems to be reproducable locally on master, once the server is mirroring enough packages, by just periodically running a curl request with a 0.2s timeout. Given it is a local server, 0.2s should be enough, and by the time it's mirroring the whole of hackage I imagine it starts to get similarly long delays.

Having a look at the code that's causing this, it seems that the `AsyncCache` is not updating as asynchronously as you'd hope, or something else is blocking waiting on this result. Given that the cache update should be running in separate `forkIO` created thread I would guess the latter. Happy to have a go at trying to fix this, but I am not entirely sure the exact cause of the issue so may need a pointer.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.