NatLabRockies / NatLabRockies/openstudio-server-helm
helm upgrade fails on web/worker Deployment spec.replicas field-manager conflict with HPA (needs --force-conflicts)
Nobody has claimed this yet.
- Dominant language
- Go Template
- Stars
- 12
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Description
Summary
helm upgrade (server-side apply, the Helm 4 default) fails against a live release with an active HorizontalPodAutoscaler because the HPA controller and Helm's chart manifest both claim ownership of spec.replicas on the web and worker Deployments:
Error: UPGRADE FAILED: ... conflict occurred while applying object openstudio-server/worker apps/v1, Kind=Deployment:
Apply failed with 1 conflict: conflict with "kube-controller-manager" with subresource "scale" using apps/v1: .spec.replicas
Root cause
The chart's Deployment templates (templates/web/web-deploy.yaml, templates/worker/worker-deploy.yaml) declare a literal replicas: field, and separate HPA objects (web HPA, worker-hpa) also manage that same field via the /scale subresource. Server-side apply tracks field managers per-field; once the HPA controller has taken ownership of .spec.replicas (which happens as soon as it scales the Deployment even once), Helm's next apply of the same field from the chart's own rendered manifest is treated as a genuine conflict rather than a benign re-declaration of the initial/desired replica count.
Impact
Any helm upgrade against a running release with an active HPA fails outright unless the operator already knows to pass --force-conflicts (not documented anywhere in this repo). This will affect essentially every routine upgrade in production once the HPAs have scaled at least once, which is very likely by the time an upgrade is performed.
Reproduced
helm upgrade openstudio-server ./openstudio-server -f openstack/values-openstack.yaml -n openstudio-server
# fails with the conflict above
helm upgrade openstudio-server ./openstudio-server -f openstack/values-openstack.yaml -n openstudio-server --force-conflicts
# succeeds
Suggested fix (pick one or combine)
- Document
--force-conflictsas a required flag forhelm upgradeinREADME.md/ deploy runbooks, alongside a short explanation of why (HPA + server-side apply field ownership). - Better: stop declaring a hardcoded
replicas:in theweb/workerDeployment templates once an HPA is enabled for that workload — e.g. only renderreplicas:when the corresponding HPA is disabled, similar to patterns used elsewhere in the chart for optional fields. This avoids the conflict at the source rather than papering over it with--force-conflicts, and matches standard Helm/HPA guidance (Deployments under HPA control conventionally omit a staticreplicasafter the first apply, or useValues.<workload>.replicas | defaultonly when HPA is off). - If (2) isn't feasible without bigger changes, at minimum wrap
helm upgradein any install/upgrade scripts (scripts/install-dry-run.shor similar) with--force-conflictsso operators/CI don't hit this by surprise.
Related
Discovered while validating PR #106 (local-image-registry) end-to-end on a live OpenStack/Azimuth cluster; unrelated to the local-registry feature itself but blocked iterating on fixes for it.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with templates/web/web-deploy.yaml and templates/worker/worker-deploy.yaml, then inspect the corresponding HPA templates and reproduce the documented helm upgrade against a release with an active HPA. Determine which proposed approach fits the chart, and verify that the upgrade path no longer fails on spec.replicas or that the required flag is documented and applied by the relevant runbook or script.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100