kiwix / kiwix/operations

Upload of mirror logs failed on Sun, 23 Aug 2026

Open
#758 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
15
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Upload of mirror logs failed on Sun, 23 Aug 2026 02:30:00 (UTC) ; matomo was down ; automatic restart by k8s was not sufficient to make it complete.

```
│ upload-mirror-logs-29790870-mst76 2026-08-23 02:30:12,320: [WARNING] DEPRECATION WARNING: passing authentication secrets via CLI options (--token-auth) is insecure because other users on the same machine can read process arguments. Use --auth-config instead. │
│ upload-mirror-logs-29790870-mst76 2026-08-23 04:06:13,139: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-mst76 2026-08-23 04:06:13,140: [INFO] Retrying request, attempt number 2 │
│ upload-mirror-logs-29790870-mst76 2026-08-23 04:08:14,717: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-mst76 2026-08-23 04:08:14,719: [INFO] Retrying request, attempt number 2 │
│ upload-mirror-logs-29790870-mst76 2026-08-23 06:06:07,944: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-mst76 2026-08-23 06:06:07,945: [INFO] Retrying request, attempt number 2 │
│ upload-mirror-logs-29790870-mst76 2026-08-23 06:07:18,076: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-mst76 2026-08-23 06:07:18,079: [INFO] Retrying request, attempt number 3 │
│ upload-mirror-logs-29790870-mst76 2026-08-23 06:08:28,238: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-mst76 2026-08-23 06:08:28,242: [INFO] Max number of attempts reached, server is unreachable! │
│ upload-mirror-logs-29790870-mst76 Fatal error: HTTP Error 504 Gateway Time-out, response: │
│ upload-mirror-logs-29790870-mst76 504 Gateway Time-out │
│ upload-mirror-logs-29790870-mst76 │
│ upload-mirror-logs-29790870-mst76

504 Gateway Time-out


│ upload-mirror-logs-29790870-mst76
nginx │
│ upload-mirror-logs-29790870-mst76 │
│ upload-mirror-logs-29790870-mst76 │
│ upload-mirror-logs-29790870-mst76 │
│ upload-mirror-logs-29790870-mst76 You can restart the import of "/var/log/nginx/access.log-20260823" from the point it failed by specifying --skip=199125 on the command line. │
│ upload-mirror-logs-29790870-mst76 │
│ stream closed: EOF for zim/upload-mirror-logs-29790870-mst76 (matomo-uploader) │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 06:08:41,454: [WARNING] DEPRECATION WARNING: passing authentication secrets via CLI options (--token-auth) is insecure because other users on the same machine can read process arguments. Use --auth-config instead. │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 08:06:05,911: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 08:06:05,913: [INFO] Retrying request, attempt number 2 │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 08:08:07,778: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 08:08:07,783: [INFO] Retrying request, attempt number 2 │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 10:05:42,825: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 10:05:42,827: [INFO] Retrying request, attempt number 2 │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 10:06:52,966: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 10:06:52,969: [INFO] Retrying request, attempt number 3 │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 10:08:03,128: [INFO] Error when connecting to Matomo: HTTP Error 504: Gateway Time-out │
│ upload-mirror-logs-29790870-ccw57 2026-08-23 10:08:03,130: [INFO] Max number of attempts reached, server is unreachable! │
│ upload-mirror-logs-29790870-ccw57 Fatal error: HTTP Error 504 Gateway Time-out, response: │
│ upload-mirror-logs-29790870-ccw57 504 Gateway Time-out │
│ upload-mirror-logs-29790870-ccw57 │
│ upload-mirror-logs-29790870-ccw57

504 Gateway Time-out


│ upload-mirror-logs-29790870-ccw57
nginx │
│ upload-mirror-logs-29790870-ccw57 │
│ upload-mirror-logs-29790870-ccw57 │
│ upload-mirror-logs-29790870-ccw57 │
│ upload-mirror-logs-29790870-ccw57 You can restart the import of "/var/log/nginx/access.log-20260823" from the point it failed by specifying --skip=190090 on the command line. │
│ upload-mirror-logs-29790870-ccw57 │
│ stream closed: EOF for zim/upload-mirror-logs-29790870-ccw57 (matomo-uploader) │

```

@rgaudin do you wanna recover from this manually or shall we just consider we will live without this data? last time I tried it was a real pain to recover.

Contributor guide

Open the contributing guide

Research direction

Start with the upload-mirror-logs Kubernetes jobs and the provided logs, then inspect the Matomo connection path and the repeated nginx HTTP 504 responses. Determine why restarting the workload does not complete the import and verify that both log files upload successfully without exhausting retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, nginx
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.