Cardinal-Cryptography / Cardinal-Cryptography/aleph-node-issues
Logs of Blackbot's Macronode servers, after the node got suspended for falling behind in the finality committee
- Dominant language
- No language data
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
### Did you read the documentation and guides?
- [x] I have inspected the documentation.
### Is there an existing issue?
- [x] I have searched the existing issues.
### Description of the problem
On 2025-04-12, we've been warned by a fellow validator that our node is falling behind in the finality committee, and that it's going to be suspended bc of that. We checked it in Grafana, and block production and all our metrics there were fine, but no metric about the finality committee were present there, so we received no warning from our prometheus > grafana > pushover monitoring system about the issue. We restarted the docker container and the machine on our non-validating server, then switched the validating server by changing the session key, then restarted the other server as well in the next session. Unfortunately we didn't backup the logs before we did the restarts, so the attached log files are all made AFTER that. :(
### Information on your setup.
Macronode1 is our primary server, its a Xenon bare metal server with a 10Gbps, unlimited connection.
Macronode2 is our failover server, its a VPS with 14vcores and 36GB RAM. We normally use it only so we have comfortable time to do maintenance on our primary server, but this time I suspect that we might have forgotten to switch the validation back to our primary server after our last maintenance, and I suspect that it was actually Macronode2 that was validating when the incident happened. Unfortunately I'm not super sure, as I was in a bit of a shock to learn that we are going to be suspended.
(By the way we are using podman instead of docker, as podman doesn't use a daemon with elevated rights to run the containers, and it is 100% compatible with docker otherwise).
### Steps to reproduce
[Macronode2_custom_files_backup_2025-04-16_00-47-28.tar.gz](https://github.com/user-attachments/files/19766792/Macronode2_custom_files_backup_2025-04-16_00-47-28.tar.gz)
[Macronode1_custom_files_backup_2025-04-16_00-42-42.tar.gz](https://github.com/user-attachments/files/19766793/Macronode1_custom_files_backup_2025-04-16_00-42-42.tar.gz)
If you guys need anything else, just tell me, I'm more than happy to help if I can!
Keep up the good job, and thank you for your work! 🙏
### Did you attach relevant logs?
- [x] I have attached logs (if relevant).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.