haskell / haskell/hackage-security
Range requests interact badly with the CDN
- Dominant language
- Haskell
- Stars
- 63
- Forks
- 56
- PR merge metrics
- No merged PRs in 30d
Description
As reported by hvr
```
[09:32:35] well, if you grab the last few bytes of 01-index.tar
[09:32:49] the request that would take 1 sec directly to hackage
[09:32:54] via the CDN takes >10 secs
[09:33:02] until the CDN has the new .tar cached
[09:33:26] probably the CDN needs to grab the full new .tar
[09:33:42] s/needs to//
[09:33:57] and that's 300MiB
[09:34:09] and since the CDN is distributed
[09:34:21] it does so multiple times for its nodes
[09:34:44] could we somehow push to the CDN?
[09:34:58] whenever the index is updated, tell the CDN "please update the .tar and .gz, now, please"
[09:35:36] I'm rather thinking of disabling the CDN for the uncompressed .tar
[09:35:53] hm
[09:36:01] with mirror selection as it currently stands
[09:36:09] that would mean that _all_ requests for updates to the central hackage server
[09:36:11] unless it does not respond
[09:36:17] in which case they will go the mirror
[09:36:36] if I do disable the CDN, then I guess we need to put some some of randomization into mirror selection?
```
```
[09:50:11] and it's not deterministic
[09:50:24] it's possible to reproduce it with curl if you're lucky
[09:50:28] (I did)
[09:50:38] ideally shortly after the tarball got updated
```
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by reproducing the range-request behavior with curl shortly after the index tarball is updated, comparing CDN and direct Hackage responses. Done means an agreed CDN or mirror-selection strategy addresses the delayed handling of the 300MiB tarball and is verified against this scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- infrastructure, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100