canonical / canonical/charm-refresh

Kubernetes pod eviction during scale down while refresh in progress temporarily sets StatefulSet partition too high

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
known limitation
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

On Kubernetes and while a refresh is in progress, if a unit's pod is deleted and re-created while that unit is being removed (for scale down), that unit may set the StatefulSet partition higher than the highest unit number, which would cause `juju refresh` to not trigger Juju events on any unit.

The partition would only be corrected when the leader unit receives its next Juju event, which might not happen until the next update-status event.

(This limitation stems from a limitation in Juju, where it is not always possible to determine if the unit that the charm code is running on is being removed without storing state about the previous events it has received. The charm-refresh implementation stores this state on the pod's ephemeral storage [for other reasons] so it is lost if the pod is deleted and re-created. Because this is an extreme edge case & the impact is only temporary, it was deemed that resolving this limitation was not worth the additional complexity [and side effects] of storing this specific state in a persistent location, separate from the other unit state.)

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files or tests. Start at the Kubernetes scale-down and refresh paths that calculate the StatefulSet partition, then trace pod recreation during unit removal. Done means the partition does not exceed the highest unit number in this edge case and refresh events resume; add or run regression coverage for that sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.