canonical / canonical/cos-proxy-operator
Action for manual refresh of rules/jobs
- Dominant language
- Python
- Stars
- 3
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
### Enhancement Proposal
There are many causes for cos-proxy to get into a state where its alert rules or scrape jobs pushed upstream to prometheus are incorrect. As the charm tries to use this stored state and base all changes off of it with incremental updates, very often we can end up with stale state. This includes duplicate rules and rules for units that no longer exist.
Many of these bugs have been fixed (thank you!) but there are always new ways for the state to diverge from the true state. I believe some of these cases are out of control of the charm, e.g. broken CMRs which required a manual `juju resolved --no-retry`, issues when migrating models, interruptions (SIGTERM) during the middle of a hook, etc... These are external issues that cannot be fixed with charm logic if the current incremental update scheme is kept.
As an operator it is frustrating to see these issues and have to either a) update the stored state by hand, or b) remove the application and redeploy it. What would be nice is to have an action `juju run cos-proxy/leader refresh-rules` that effectively wipes the rules, runs through each of the downstream relations and rebuilds the full set.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the charm's action definitions and the handlers that process downstream relations; the issue names no files or tests, so repository exploration is required. Done means a leader-only refresh-rules action clears the stored rules and jobs, rebuilds them from every downstream relation, and leaves Prometheus state synchronized.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100