thoughtbot / thoughtbot/flightdeck
Node running prometheus becomes unavailable
Nobody has claimed this yet.
- Dominant language
- HCL
- Stars
- 100
- Forks
- 11
- Avg merge
- 7h 37m
- Merged PRs (30d)
- 3
Description
This has been something happening on Albaik recently, the node that prometheus is running on becomes unavailable, which causes issues for monitoring and application level autoscaling.
The current fix is to manually intervene and replace the node, as documented [here].(https://github.com/thoughtbot/mission-control-platform/blob/main/aws/src/debug/cluster-errors.md#unreachable-nodes)
This issue is to track debugging the process and figuring out what happens, and resolve it automatically.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with aws/src/debug/cluster-errors.md#unreachable-nodes, which documents the current manual response to unreachable nodes. Trace how the Prometheus node becomes unavailable and define an automated replacement path; done means the failure is understood and the node can be recovered without manual intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, prometheus, terraform
- Domain
- cloud, infrastructure, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100