Monitor via logs

At the moment, we collect all apps logs in grafana but it's mostly used for forensic.

We do have a few alerts in place that triggers notification to slack (`#infra-alerts`):

- `HTTP/1.1" 5` in the logs (exceptions for `502`, `503`, cms and wiki)
- `ERROR` in `cardshop` namespace
- `Pushing IP changes to Wasabi failed` in `zimfarm/api`

Due to the poor quality of the notification (message is unclear about which service, link is not useful, there are multiple alerts for the same cause) and the fact that it's polluted by misbehaving services (ex: matomo), those are generally silenced and not acted upon.

After discussing the matter with @kelson42, we think it's time we leverage the logs for monitoring:

- HTTP 50x alerts should all be handled manually
- `ERROR` level in log should all be handled manually

We acknowledge that this will be unmanageable at first but we will put in the effort to fix each service so that the errors reported are actual errors and meaningful. _External_ services over which we have no control may be added as exceptions.

Once done, how quick we should react to those is to be determined and doesn't have to be the same for each service.

First task is probably to figure out a way to properly report those from grafana to slack in an actionable fashion.