canonical / canonical/github-runner-operators

fix(garm): make application removal safe during relation teardown

Open
#332 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
2
Avg merge
1d 13h
Merged PRs (30d)
46

Description

## Bug description

Removing the GARM application can race with Juju relation teardown. Relation-departed/broken hooks re-enter the normal reconcile path after relation data has already disappeared, causing the removal workflow to fail before or during application teardown.

## Evidence from production removal test

During a targeted removal of `github-runner-garm`:

- `ingress-relation-departed` failed with the relation snapshot warning `"app" expected but not received`.
- `garm-configurator-relation-broken` failed repeatedly with the same snapshot warning and `secret_get` errors while reconstructing charm state.
- `postgresql-relation-broken` and `debug-ssh-relation-broken` also observed missing relation snapshot data.
- The unit remained in error/awaiting error resolution until failed hooks were marked resolved without retry.
- A remaining cross-model debug-ssh relation had to be force-removed before the dying application disappeared.

Relevant Juju bug reference emitted by the relation libraries: https://bugs.launchpad.net/juju/+bug/1960934

## Expected behavior

The remove path should not run normal relation-driven reconciliation after teardown has started. Relation-departed/broken handlers should tolerate missing snapshot data during application removal, and the dedicated `on.remove` cleanup should run while GARM and its database/API are still available.

## Suggested direction

- Gate or short-circuit relation handlers when the application is dying/removing.
- Ensure cleanup ordering does not depend on relation data that Juju may remove first.
- Add an integration test for application removal with all current relations, including cross-model relations.
- Preserve the GARM encryption secret or otherwise define database/secret lifecycle for application recreation.

## Environment

- PS7 production Kubernetes model: `k8s-pfe-ps7-prod-github-runner-prod`
- GARM revision: 104
- Test ticket: ISD-6082

Contributor guide

Open the contributing guide

Research direction

Start by tracing the relation-departed and relation-broken handlers alongside the dedicated on.remove cleanup path, then inspect how application-dying state and relation snapshots are handled. Add the requested integration test covering removal with current and cross-model relations, and verify that hooks tolerate missing data while cleanup completes and the application is removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
40/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.