Cardinal-Cryptography / Cardinal-Cryptography/aleph-node-issues

Logs of Blackbot's Macronode servers, after the node got suspended for falling behind in the finality committee

Offen
#19 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Keine Sprachdaten
Sterne
2
Forks
0
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

### Did you read the documentation and guides?

- [x] I have inspected the documentation.

### Is there an existing issue?

- [x] I have searched the existing issues.

### Description of the problem

On 2025-04-12, we've been warned by a fellow validator that our node is falling behind in the finality committee, and that it's going to be suspended bc of that. We checked it in Grafana, and block production and all our metrics there were fine, but no metric about the finality committee were present there, so we received no warning from our prometheus > grafana > pushover monitoring system about the issue. We restarted the docker container and the machine on our non-validating server, then switched the validating server by changing the session key, then restarted the other server as well in the next session. Unfortunately we didn't backup the logs before we did the restarts, so the attached log files are all made AFTER that. :(

### Information on your setup.

Macronode1 is our primary server, its a Xenon bare metal server with a 10Gbps, unlimited connection.
Macronode2 is our failover server, its a VPS with 14vcores and 36GB RAM. We normally use it only so we have comfortable time to do maintenance on our primary server, but this time I suspect that we might have forgotten to switch the validation back to our primary server after our last maintenance, and I suspect that it was actually Macronode2 that was validating when the incident happened. Unfortunately I'm not super sure, as I was in a bit of a shock to learn that we are going to be suspended.
(By the way we are using podman instead of docker, as podman doesn't use a daemon with elevated rights to run the containers, and it is 100% compatible with docker otherwise).

### Steps to reproduce

[Macronode2_custom_files_backup_2025-04-16_00-47-28.tar.gz](https://github.com/user-attachments/files/19766792/Macronode2_custom_files_backup_2025-04-16_00-47-28.tar.gz)
[Macronode1_custom_files_backup_2025-04-16_00-42-42.tar.gz](https://github.com/user-attachments/files/19766793/Macronode1_custom_files_backup_2025-04-16_00-42-42.tar.gz)

If you guys need anything else, just tell me, I'm more than happy to help if I can!
Keep up the good job, and thank you for your work! 🙏

### Did you attach relevant logs?

- [x] I have attached logs (if relevant).

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Beginne mit der Untersuchung der angehängten Backups von Macronode1 und Macronode2 für benutzerdefinierte Dateien sowie der Logs nach dem Neustart. Überprüfe anschließend die Monitoring-Konfiguration von Prometheus und Grafana. Ermittle, ob der Zustand des Finality Committees erfasst wird und ob der bestehende Pushover-Alarmweg dies erkennen kann; abgeschlossen ist die Aufgabe mit einer bestätigten Diagnose oder einer klar abgegrenzten Monitoring-Änderung.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
docker, grafana, prometheus
Bereich
infrastructure, observability-sre
Issue-Typ
Bug
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.