07th-mod / 07th-mod/python-patcher

Installer Download / URL checker problems / Strange server behavior with partial requests

Abierto
#165 4 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
JavaScript
Estrellas
214
Forks
12
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

When investigating why the URL checker didn't work, @tellowkrinkle discovered that the server won't always send the correct response if you only request part of a file.

It looks like every second response will give a different response, alternating between HTTP 200 (the whole file), and HTTP 206 (partial response):

```cmd
C:\Users\drojf>curl --head "https://07th-mod.com/rikachama/graphics/Watanagashi-Graphics.7z" -H "Range: bytes=0-1023"
HTTP/1.1 200 OK
Date: Sun, 19 Sep 2021 01:41:37 GMT
Content-Type: application/x-7z-compressed
Content-Length: 1006748838
Connection: keep-alive
last-modified: Mon, 12 Oct 2020 23:09:17 GMT
etag: "5f84e21d-3c01c4a6"
Cache-Control: max-age=3600
CF-Cache-Status: MISS
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=mzUHRUmntFmMAdOY%2FT9om2cIKrUpjUPKU%2FNdqCofBmZ0ex1wMttuDkfcONIyBCdTUQJ%2FoipAQlY1aaNiDPwqxjJSl7%2BP1213rdqqMwO5PtPJD%2B1KnQeSkVecs2jLbeY%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 690f1cd88c5c16c5-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

C:\Users\drojf>curl --head "https://07th-mod.com/rikachama/graphics/Watanagashi-Graphics.7z" -H "Range: bytes=0-1023"
HTTP/1.1 206 Partial Content
Date: Sun, 19 Sep 2021 01:41:39 GMT
Content-Type: application/x-7z-compressed
Content-Length: 1024
Connection: keep-alive
last-modified: Mon, 12 Oct 2020 23:09:17 GMT
etag: "5f84e21d-3c01c4a6"
content-range: bytes 0-1023/1006748838
Cache-Control: max-age=3600
CF-Cache-Status: MISS
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=PCimvFfIu0rza0%2FAeyxog9j3bGk2q6r7Tlm7u44XnZcBlKmcL6HOS6lwCstnij1E7isipUtlHQYiQi5rSmoph%2BoTVm7UCu5Bo280PMFUWzlVZr120rQEsCnu1lpm0x8%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 690f1ce28a1a16b9-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
```

I tested doing range requests on another website (downloading the Ubuntu ISO), and that website doesn't have this problem - it always returns 206.

Now this may seem like it would break the installer's "resume downloads" capability, but aria2c always first does a normal request, followed by a range request. When you do the requests this way, the server will always give a correct range response on the second request.

```cmd
09/19 13:16:25 [INFO] CUID#7 - Requesting:
GET /rikachama/graphics/Watanagashi-Graphics.7z HTTP/1.1
User-Agent: aria2/1.35.0
Accept: */*,application/metalink4+xml,application/metalink+xml
Host: 07th-mod.com
Want-Digest: SHA-512;q=1, SHA-256;q=1, SHA;q=0.1

[#9392da 0B/0B CN:1 DL:0B]
09/19 13:16:26 [INFO] CUID#7 - Response received:
HTTP/1.1 200 OK
Date: Sun, 19 Sep 2021 03:16:24 GMT
Content-Type: application/x-7z-compressed
Content-Length: 1006748838
Connection: keep-alive
last-modified: Mon, 12 Oct 2020 23:09:17 GMT
etag: "5f84e21d-3c01c4a6"
Cache-Control: max-age=3600
CF-Cache-Status: MISS
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=e13MFXOE%2FDL0pPprAmHZ%2BvK2cexqyTqMxP5j7X29KHXDlP6CnCWCXGLM25HYUQ57tGbLnhhyYQ64F0pq6%2B0ScohBZJLoufugicKtDdmvSM%2BTw1yOGf6vUXiqQsWzYcc%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 690fa7ad9e1b6a1a-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

09/19 13:16:26 [INFO] CUID#8 - Requesting:
GET /rikachama/graphics/Watanagashi-Graphics.7z HTTP/1.1
User-Agent: aria2/1.35.0
Accept: */*
Host: 07th-mod.com
Range: bytes=241631232-1006748837
Want-Digest: SHA-512;q=1, SHA-256;q=1, SHA;q=0.1

[#9392da 230MiB/0.9GiB(24%) CN:1 DL:0B]
09/19 13:16:27 [INFO] CUID#8 - Response received:
HTTP/1.1 206 Partial Content
Date: Sun, 19 Sep 2021 03:16:25 GMT
Content-Type: application/x-7z-compressed
Content-Length: 765117606
Connection: keep-alive
last-modified: Mon, 12 Oct 2020 23:09:17 GMT
etag: "5f84e21d-3c01c4a6"
content-range: bytes 241631232-1006748837/1006748838
Cache-Control: max-age=3600
CF-Cache-Status: MISS
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=pse79IiOe%2BsMrwDhjQICTw4v0c8bmSBkby2xudgF7bC9AgpUaT7orsMmTi8n6%2BxfVuuvQ0N0O%2F%2BavyfBTCvIIVDhhHjs8lKr6dtPFHy17iw3fAendzRlQsBs5auBnNQ%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 690fa7b64fa516e1-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
```

This means that our installer's resume functionality is unaffected by this behavior. Therefore, I don't think we need to investigate/fix this issue.

Tellowkrinkle has put a workaround for the URL checker (along with quite a few other changes) on the `fix_validator_timeout` branch, which should be ready to merge already. I'll close this issue once that is merged.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

El issue describe un comportamiento del servidor donde las solicitudes parciales de rango HTTP alternan entre respuestas 200 y 206. La solución alternativa ya está en la rama `fix_validator_timeout`. Para entender el problema, examina el código del verificador de URL que realiza las solicitudes de rango, y observa cómo la rama lo modifica. La solución probablemente implica manejar ambos códigos de respuesta o asegurar que una solicitud completa preceda a la solicitud de rango.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Área
backend, cli
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.