ChainSafe / ChainSafe/forest-iac
Revive long-running nodes (mainnet and calibnet)
- Dominant language
- Ruby
- Stars
- 11
- Forks
- 6
- Avg merge
- 4h 52m
- Merged PRs (30d)
- 3
Description
**Issue summary**
Forest had some hiccups recently, likely due to the new-ish chain follower implementation. See https://github.com/ChainSafe/forest/issues/6042 as an example.
There is already [code](https://github.com/ChainSafe/forest-iac/blob/main/tf-managed/live/environments/dev/applications/forest-mainnet/terragrunt.hcl) for deployment of such nodes (to be reviewed, some time has passed since it was last updated).
Still, an essential part is missing, being monitoring & alerting. There was supposed to be a [follow up](https://github.com/ChainSafe/forest-iac/issues/413) after refactoring this piece, but it never happened (for various reasons).
What needs to be done:
- [ ] Review the current deployment logic. Perhaps it can be simplified with new Forest features. Is there anything else missing? For example, it'd be great to create coredumps on panics.
- [ ] Decide on the monitoring & alerting solution. We already have some code in git history for integrating with New Relic, but perhaps it's better to use ChainSafe's Grafana, if possible.
- [ ] Implement monitoring & alerting based on the previous point. Discuss healthchecks with the team. Set up relevant alerts (so that the team is informed on real failures; we want to avoid false positives). Node's getting stuck, crash, OOM - likely a good candidate for such alerts.
- [ ] Decide on the schedule for handling the alerts. Polevik might be a good person to put this duty on.
**Other information and links**
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.