Cardinal-Cryptography / Cardinal-Cryptography/aleph-node-issues

Logs of Blackbot's Macronode servers, after the node got suspended for falling behind in the finality committee

Open
#19 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

### Did you read the documentation and guides?

- [x] I have inspected the documentation.

### Is there an existing issue?

- [x] I have searched the existing issues.

### Description of the problem

On 2025-04-12, we've been warned by a fellow validator that our node is falling behind in the finality committee, and that it's going to be suspended bc of that. We checked it in Grafana, and block production and all our metrics there were fine, but no metric about the finality committee were present there, so we received no warning from our prometheus > grafana > pushover monitoring system about the issue. We restarted the docker container and the machine on our non-validating server, then switched the validating server by changing the session key, then restarted the other server as well in the next session. Unfortunately we didn't backup the logs before we did the restarts, so the attached log files are all made AFTER that. :(

### Information on your setup.

Macronode1 is our primary server, its a Xenon bare metal server with a 10Gbps, unlimited connection.
Macronode2 is our failover server, its a VPS with 14vcores and 36GB RAM. We normally use it only so we have comfortable time to do maintenance on our primary server, but this time I suspect that we might have forgotten to switch the validation back to our primary server after our last maintenance, and I suspect that it was actually Macronode2 that was validating when the incident happened. Unfortunately I'm not super sure, as I was in a bit of a shock to learn that we are going to be suspended.
(By the way we are using podman instead of docker, as podman doesn't use a daemon with elevated rights to run the containers, and it is 100% compatible with docker otherwise).

### Steps to reproduce

[Macronode2_custom_files_backup_2025-04-16_00-47-28.tar.gz](https://github.com/user-attachments/files/19766792/Macronode2_custom_files_backup_2025-04-16_00-47-28.tar.gz)
[Macronode1_custom_files_backup_2025-04-16_00-42-42.tar.gz](https://github.com/user-attachments/files/19766793/Macronode1_custom_files_backup_2025-04-16_00-42-42.tar.gz)

If you guys need anything else, just tell me, I'm more than happy to help if I can!
Keep up the good job, and thank you for your work! 🙏

### Did you attach relevant logs?

- [x] I have attached logs (if relevant).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.