haskell / haskell/hackage-server

hackage-mirror stops on server-500 error

Open
#1,023 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
467
Forks
225
PR merge metrics
No merged PRs in 30d

Description

When running hackage-mirror, the client eventually hits `hermes-1.3.4.3`, which doesn't seem to have a file on hackage. The mirror client then stops mirroring for half an hour (due to `--continuous`) and then tries again, failing in the same way:

```
$ hackage-mirror servers.cfg --continuous -vvv
Selected mirror http://hackage.haskell.org
Downloading timestamp
Sending:
GET /timestamp.json HTTP/1.1
Host: hackage.haskell.org
User-Agent: hackage-mirror/0.5.1

Creating new connection to hackage.haskell.org
Received:
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Cache-Control: public, no-transform, max-age=60
Content-MD5: U+wGwY5oXtD8pDDn/hX6dw==
ETag: "53ec06c18e685ed0fca430e7fe15fa77"
Content-Length: 464
Accept-Ranges: bytes
Date: Tue, 22 Feb 2022 18:36:16 GMT
Via: 1.1 varnish
Age: 0
Connection: keep-alive
X-Served-By: cache-lax10658-LGB
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1645554976.957252,VS0,VE97
Vary: Accept-Encoding

Waiting to acquire cache lock on /runtime/mirror-cache/hackage/hackage-security-lock
Acquired cache lock on /runtime/mirror-cache/hackage/hackage-security-lock
Released cache lock on /runtime/mirror-cache/hackage/hackage-security-lock
No cached index available for target repo
downloading http://admin:...@localhost:8080/packages/index.tar.gz to
mirror-cache/mirror/00-index.tar.gz
Sending:
GET /packages/index.tar.gz HTTP/1.1
Host: localhost:8080
User-Agent: hackage-mirror/0.5.1
If-None-Match: "7896f4cd4a536563d5509acff1bdf0ff"
Authorization: Basic YWRtaW46YWRtaW4=

Creating new connection to admin:admin@localhost:8080
Received:
HTTP/1.1 304 Not Modified
Cache-Control: public, no-transform, max-age=300
Connection: Keep-Alive
Date: Tue, 22 Feb 2022 18:36:22 GMT
ETag: "7896f4cd4a536563d5509acff1bdf0ff"
Server: Happstack/7.7.1.2

mirror-cache/mirror/00-index.tar.gz unchanged with ETag
7896f4cd4a536563d5509acff1bdf0ff
87303 packages to mirror. Ignoring 15849 package(s) that cannot be mirrored
(for details see mirror-cache/missing-hackage and
mirror-cache/unmirrorable-hackage-mirror)
mirroring hermes-1.3.4.3
Selected mirror http://hackage.haskell.org
Downloading package hermes-1.3.4.3
Sending:
GET /package/hermes-1.3.4.3.tar.gz HTTP/1.1
Host: hackage.haskell.org
User-Agent: hackage-mirror/0.5.1

Creating new connection to hackage.haskell.org
Received:
HTTP/1.1 301 Moved Permanently
Server: nginx/1.18.0 (Ubuntu)
Content-Type: text/plain; charset=UTF-8
Location: /package/hermes-1.3.4.3/hermes-1.3.4.3.tar.gz
Content-Length: 0
Accept-Ranges: bytes
Date: Tue, 22 Feb 2022 18:36:26 GMT
Via: 1.1 varnish
Age: 2382
Connection: keep-alive
X-Served-By: cache-lax10656-LGB
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1645554987.548999,VS0,VE1

301 - redirect
Redirecting to
http://hackage.haskell.org/package/hermes-1.3.4.3/hermes-1.3.4.3.tar.gz ...
Sending:
GET /package/hermes-1.3.4.3/hermes-1.3.4.3.tar.gz HTTP/1.1
Host: hackage.haskell.org
User-Agent: hackage-mirror/0.5.1

Creating new connection to hackage.haskell.org
Received:
HTTP/1.1 500 Internal Server Error
Server: nginx/1.18.0 (Ubuntu)
Content-Type: text/html; charset=UTF-8
Fastly-Restarts: 1
Transfer-Encoding: chunked
Accept-Ranges: bytes
Date: Tue, 22 Feb 2022 18:36:26 GMT
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: cache-lax10679-LGB
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1645554987.648724,VS0,VE166
Content-Length: 445

Warning: Failed to download package hermes-1.3.4.3,
Unexpected response (5,0,0)for http://hackage.haskell.orgpackage/hermes-1.3.4.3.tar.gz
Abandoning this mirroring attempt.
Next try will be in 33 minutes, at 19:09 UTC
```

Current output from http://hackage.haskell.org/package/hermes-1.3.4.3.tar.gz (note that the URL listed in the "unexpected response" above is missing a `/`):

```
Happstack 7.7.1
Something went wrong here
Internal server error
Everything has stopped

The error was "/home/hackage/hackage.prod/state/blobs/a4/a40900b9b0da3715595660c9ede6df77: openBinaryFile: does not exist (No such file or directory)"
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.