haskell / haskell/hackage-security

Range requests interact badly with the CDN

Open
#104 1 comment 0 reactions 0 assignees View on GitHub
hackage-security
Dominant language
Haskell
Stars
63
Forks
56
PR merge metrics
No merged PRs in 30d

Description

As reported by hvr

```
[09:32:35] well, if you grab the last few bytes of 01-index.tar
[09:32:49] the request that would take 1 sec directly to hackage
[09:32:54] via the CDN takes >10 secs
[09:33:02] until the CDN has the new .tar cached
[09:33:26] probably the CDN needs to grab the full new .tar
[09:33:42] s/needs to//
[09:33:57] and that's 300MiB
[09:34:09] and since the CDN is distributed
[09:34:21] it does so multiple times for its nodes
[09:34:44] could we somehow push to the CDN?
[09:34:58] whenever the index is updated, tell the CDN "please update the .tar and .gz, now, please"
[09:35:36] I'm rather thinking of disabling the CDN for the uncompressed .tar
[09:35:53] hm
[09:36:01] with mirror selection as it currently stands
[09:36:09] that would mean that _all_ requests for updates to the central hackage server
[09:36:11] unless it does not respond
[09:36:17] in which case they will go the mirror
[09:36:36] if I do disable the CDN, then I guess we need to put some some of randomization into mirror selection?
```

```
[09:50:11] and it's not deterministic
[09:50:24] it's possible to reproduce it with curl if you're lucky
[09:50:28] (I did)
[09:50:38] ideally shortly after the tarball got updated
```

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by reproducing the range-request behavior with curl shortly after the index tarball is updated, comparing CDN and direct Hackage responses. Done means an agreed CDN or mirror-selection strategy addresses the delayed handling of the 300MiB tarball and is verified against this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
infrastructure, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.