canonical / canonical/github-runner-operators
fix(garm): make application removal safe during relation teardown
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 46
Description
## Bug description
Removing the GARM application can race with Juju relation teardown. Relation-departed/broken hooks re-enter the normal reconcile path after relation data has already disappeared, causing the removal workflow to fail before or during application teardown.
## Evidence from production removal test
During a targeted removal of `github-runner-garm`:
- `ingress-relation-departed` failed with the relation snapshot warning `"app" expected but not received`.
- `garm-configurator-relation-broken` failed repeatedly with the same snapshot warning and `secret_get` errors while reconstructing charm state.
- `postgresql-relation-broken` and `debug-ssh-relation-broken` also observed missing relation snapshot data.
- The unit remained in error/awaiting error resolution until failed hooks were marked resolved without retry.
- A remaining cross-model debug-ssh relation had to be force-removed before the dying application disappeared.
Relevant Juju bug reference emitted by the relation libraries: https://bugs.launchpad.net/juju/+bug/1960934
## Expected behavior
The remove path should not run normal relation-driven reconciliation after teardown has started. Relation-departed/broken handlers should tolerate missing snapshot data during application removal, and the dedicated `on.remove` cleanup should run while GARM and its database/API are still available.
## Suggested direction
- Gate or short-circuit relation handlers when the application is dying/removing.
- Ensure cleanup ordering does not depend on relation data that Juju may remove first.
- Add an integration test for application removal with all current relations, including cross-model relations.
- Preserve the GARM encryption secret or otherwise define database/secret lifecycle for application recreation.
## Environment
- PS7 production Kubernetes model: `k8s-pfe-ps7-prod-github-runner-prod`
- GARM revision: 104
- Test ticket: ISD-6082
Contributor guide
Research direction
Start by tracing the relation-departed and relation-broken handlers alongside the dedicated on.remove cleanup path, then inspect how application-dying state and relation snapshots are handled. Add the requested integration test covering removal with current and cross-model relations, and verify that hooks tolerate missing data while cleanup completes and the application is removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 40/100