canonical / canonical/charm-refresh
Kubernetes second highest number unit does not start workload if highest number unit is leader and raises uncaught exception in event where refresh started
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
On Kubernetes, if the highest number unit
- is leader,
- successfully starts the application's refresh (because
- the refresh is a rollback,
- the automatic checks [
- that the deployed workload OCI image hash matches the hash pinned in the charm code
- that the refresh is compatible
- pre-refresh health checks & preparations
] succeeded,
- or the `force-refresh-start` action was successfully run
),
- sets `next_unit_allowed_to_refresh = True` in the same Juju event where the application's refresh was started,
- and raises an uncaught exception in the same Juju event
then the second highest number unit will be refreshed but its workload will not start until the highest number unit stops raising an uncaught exception.
Mitigating this limitation would require adding a dependency on 1+ other units being healthy to trigger a Juju event on the highest number unit. That would impair the robustness of refreshes/rollbacks, which do not currently have this dependency. In particular, the extra dependency would prevent recovery in these situations:
- All units suddenly start raising an uncaught exception because of a change to the environment and the only way to fix the issue is by refreshing the charm
- Kubernetes pod evictions or charm container restarts while a refresh is in progress causes all non-refreshed units to go offline because of https://bugs.launchpad.net/juju/+bug/2073506
Contributor guide
No contributing guide indexed for this repository
Research direction
The report names no files or tests; start by tracing the Kubernetes refresh coordination and the Juju event path that sets next_unit_allowed_to_refresh. Reproduce the highest-unit exception scenario and inspect how the second-highest unit starts its workload. Done requires a resolution that handles the failure without compromising refresh and rollback recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100