canonical / canonical/charm-refresh

Kubernetes second highest number unit does not start workload if highest number unit is leader and raises uncaught exception in event where refresh started

Open
#9 0 comments 1 reaction 0 assignees View on GitHub
known limitation
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

On Kubernetes, if the highest number unit

- is leader,
- successfully starts the application's refresh (because
- the refresh is a rollback,
- the automatic checks [
- that the deployed workload OCI image hash matches the hash pinned in the charm code
- that the refresh is compatible
- pre-refresh health checks & preparations

] succeeded,
- or the `force-refresh-start` action was successfully run

),
- sets `next_unit_allowed_to_refresh = True` in the same Juju event where the application's refresh was started,
- and raises an uncaught exception in the same Juju event

then the second highest number unit will be refreshed but its workload will not start until the highest number unit stops raising an uncaught exception.

Mitigating this limitation would require adding a dependency on 1+ other units being healthy to trigger a Juju event on the highest number unit. That would impair the robustness of refreshes/rollbacks, which do not currently have this dependency. In particular, the extra dependency would prevent recovery in these situations:
- All units suddenly start raising an uncaught exception because of a change to the environment and the only way to fix the issue is by refreshing the charm
- Kubernetes pod evictions or charm container restarts while a refresh is in progress causes all non-refreshed units to go offline because of https://bugs.launchpad.net/juju/+bug/2073506

Contributor guide

No contributing guide indexed for this repository

Research direction

The report names no files or tests; start by tracing the Kubernetes refresh coordination and the Juju event path that sets next_unit_allowed_to_refresh. Reproduce the highest-unit exception scenario and inspect how the second-highest unit starts its workload. Done requires a resolution that handles the failure without compromising refresh and rollback recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.