07th-mod / 07th-mod/python-patcher

Installer Download / URL checker problems / Strange server behavior with partial requests

Offen
#165 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
JavaScript
Sterne
214
Forks
12
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

When investigating why the URL checker didn't work, @tellowkrinkle discovered that the server won't always send the correct response if you only request part of a file.

It looks like every second response will give a different response, alternating between HTTP 200 (the whole file), and HTTP 206 (partial response):

```cmd
C:\Users\drojf>curl --head "https://07th-mod.com/rikachama/graphics/Watanagashi-Graphics.7z" -H "Range: bytes=0-1023"
HTTP/1.1 200 OK
Date: Sun, 19 Sep 2021 01:41:37 GMT
Content-Type: application/x-7z-compressed
Content-Length: 1006748838
Connection: keep-alive
last-modified: Mon, 12 Oct 2020 23:09:17 GMT
etag: "5f84e21d-3c01c4a6"
Cache-Control: max-age=3600
CF-Cache-Status: MISS
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=mzUHRUmntFmMAdOY%2FT9om2cIKrUpjUPKU%2FNdqCofBmZ0ex1wMttuDkfcONIyBCdTUQJ%2FoipAQlY1aaNiDPwqxjJSl7%2BP1213rdqqMwO5PtPJD%2B1KnQeSkVecs2jLbeY%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 690f1cd88c5c16c5-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

C:\Users\drojf>curl --head "https://07th-mod.com/rikachama/graphics/Watanagashi-Graphics.7z" -H "Range: bytes=0-1023"
HTTP/1.1 206 Partial Content
Date: Sun, 19 Sep 2021 01:41:39 GMT
Content-Type: application/x-7z-compressed
Content-Length: 1024
Connection: keep-alive
last-modified: Mon, 12 Oct 2020 23:09:17 GMT
etag: "5f84e21d-3c01c4a6"
content-range: bytes 0-1023/1006748838
Cache-Control: max-age=3600
CF-Cache-Status: MISS
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=PCimvFfIu0rza0%2FAeyxog9j3bGk2q6r7Tlm7u44XnZcBlKmcL6HOS6lwCstnij1E7isipUtlHQYiQi5rSmoph%2BoTVm7UCu5Bo280PMFUWzlVZr120rQEsCnu1lpm0x8%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 690f1ce28a1a16b9-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
```

I tested doing range requests on another website (downloading the Ubuntu ISO), and that website doesn't have this problem - it always returns 206.

Now this may seem like it would break the installer's "resume downloads" capability, but aria2c always first does a normal request, followed by a range request. When you do the requests this way, the server will always give a correct range response on the second request.

```cmd
09/19 13:16:25 [INFO] CUID#7 - Requesting:
GET /rikachama/graphics/Watanagashi-Graphics.7z HTTP/1.1
User-Agent: aria2/1.35.0
Accept: */*,application/metalink4+xml,application/metalink+xml
Host: 07th-mod.com
Want-Digest: SHA-512;q=1, SHA-256;q=1, SHA;q=0.1

[#9392da 0B/0B CN:1 DL:0B]
09/19 13:16:26 [INFO] CUID#7 - Response received:
HTTP/1.1 200 OK
Date: Sun, 19 Sep 2021 03:16:24 GMT
Content-Type: application/x-7z-compressed
Content-Length: 1006748838
Connection: keep-alive
last-modified: Mon, 12 Oct 2020 23:09:17 GMT
etag: "5f84e21d-3c01c4a6"
Cache-Control: max-age=3600
CF-Cache-Status: MISS
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=e13MFXOE%2FDL0pPprAmHZ%2BvK2cexqyTqMxP5j7X29KHXDlP6CnCWCXGLM25HYUQ57tGbLnhhyYQ64F0pq6%2B0ScohBZJLoufugicKtDdmvSM%2BTw1yOGf6vUXiqQsWzYcc%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 690fa7ad9e1b6a1a-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

09/19 13:16:26 [INFO] CUID#8 - Requesting:
GET /rikachama/graphics/Watanagashi-Graphics.7z HTTP/1.1
User-Agent: aria2/1.35.0
Accept: */*
Host: 07th-mod.com
Range: bytes=241631232-1006748837
Want-Digest: SHA-512;q=1, SHA-256;q=1, SHA;q=0.1

[#9392da 230MiB/0.9GiB(24%) CN:1 DL:0B]
09/19 13:16:27 [INFO] CUID#8 - Response received:
HTTP/1.1 206 Partial Content
Date: Sun, 19 Sep 2021 03:16:25 GMT
Content-Type: application/x-7z-compressed
Content-Length: 765117606
Connection: keep-alive
last-modified: Mon, 12 Oct 2020 23:09:17 GMT
etag: "5f84e21d-3c01c4a6"
content-range: bytes 241631232-1006748837/1006748838
Cache-Control: max-age=3600
CF-Cache-Status: MISS
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=pse79IiOe%2BsMrwDhjQICTw4v0c8bmSBkby2xudgF7bC9AgpUaT7orsMmTi8n6%2BxfVuuvQ0N0O%2F%2BavyfBTCvIIVDhhHjs8lKr6dtPFHy17iw3fAendzRlQsBs5auBnNQ%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 690fa7b64fa516e1-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
```

This means that our installer's resume functionality is unaffected by this behavior. Therefore, I don't think we need to investigate/fix this issue.

Tellowkrinkle has put a workaround for the URL checker (along with quite a few other changes) on the `fix_validator_timeout` branch, which should be ready to merge already. I'll close this issue once that is merged.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Das Issue beschreibt ein Serververhalten, bei dem partielle HTTP-Range-Requests zwischen 200- und 206-Antworten wechseln. Der Workaround befindet sich bereits auf dem Branch `fix_validator_timeout`. Um das Problem zu verstehen, untersuche den URL-Checker-Code, der Range-Requests durchführt, und sieh dir an, wie der Branch ihn modifiziert. Die Lösung beinhaltet wahrscheinlich die Handhabung beider Antwortcodes oder das Sicherstellen, dass ein vollständiger Request dem Range-Request vorausgeht.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Bereich
backend, cli
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.