canonical / canonical/cos-proxy-operator

Action for manual refresh of rules/jobs

Open
#209 3 comments 0 reactions 0 assignees View on GitHub
Status: Triage Type: Enhancement
Dominant language
Python
Stars
3
Forks
12
PR merge metrics
No merged PRs in 30d

Description

### Enhancement Proposal

There are many causes for cos-proxy to get into a state where its alert rules or scrape jobs pushed upstream to prometheus are incorrect. As the charm tries to use this stored state and base all changes off of it with incremental updates, very often we can end up with stale state. This includes duplicate rules and rules for units that no longer exist.

Many of these bugs have been fixed (thank you!) but there are always new ways for the state to diverge from the true state. I believe some of these cases are out of control of the charm, e.g. broken CMRs which required a manual `juju resolved --no-retry`, issues when migrating models, interruptions (SIGTERM) during the middle of a hook, etc... These are external issues that cannot be fixed with charm logic if the current incremental update scheme is kept.

As an operator it is frustrating to see these issues and have to either a) update the stored state by hand, or b) remove the application and redeploy it. What would be nice is to have an action `juju run cos-proxy/leader refresh-rules` that effectively wipes the rules, runs through each of the downstream relations and rebuilds the full set.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the charm's action definitions and the handlers that process downstream relations; the issue names no files or tests, so repository exploration is required. Done means a leader-only refresh-rules action clears the stored rules and jobs, rebuilds them from every downstream relation, and leaves Prometheus state synchronized.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.