kiwix / kiwix/operations

What's a mirroring incident?

Open
#714 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
15
Forks
5
PR merge metrics
No merged PRs in 30d

Description

We are now checking responsiveness about every minute for all our (13) mirrors.

If the probe doesn't respond or responds incorrectly (be it a bad certs, non-200 HTTP, timeout, etc), we change its status in the DB.
With the status set to `false`, `mod_mirrorbrain` doesn't consider this mirror when forwarding requests.
When the mirror comes back (can be the next minute), it's status is changed to `true` and `mod_mirrorbrain` considers it.

It's all automatic and working fine but we have no data regarding the consequences.

- Maybe there's an rsync problem and we're the only one serving a large ZIM
- Maybe mirror(s) serving part of the work are not working and we're offloading the traffic elsewhere.

We our current configuration, the Kiwix mirror is used as fallback. It's good in that no user is left behind but it means that any problem in the mirroring infra increases traffic on this single (and HTTP-slow) server. It's OK if we're aware of it and can act upon a resolution, but we're not.

So **we need to define what qualifies as an incident that should be reported**.

Keeping in mind that a mirror:

- does not declare what it _wants_ to mirror
- can be serving only an IP prefix or ASN (we don't do this ATM), or a Region or a Country or a list of countries

See https://github.com/kiwix/minibrain/issues/1

Contributor guide

Open the contributing guide

Research direction

Start with the linked minibrain issue and the mirror behavior described here, including health-check status changes and Kiwix fallback traffic. Done means an agreed definition of reportable mirroring incidents that accounts for partial mirror scope such as regions, countries, prefixes, ASNs, and declared coverage.

Written by the indexing model from the issue text.

Assessment

Domain
infrastructure, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.