NatLabRockies / NatLabRockies/openstudio-server-helm

helm upgrade fails on web/worker Deployment spec.replicas field-manager conflict with HPA (needs --force-conflicts)

Open
#110 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go Template
Stars
12
Forks
24
PR merge metrics
No merged PRs in 30d

Description

Summary

helm upgrade (server-side apply, the Helm 4 default) fails against a live release with an active HorizontalPodAutoscaler because the HPA controller and Helm's chart manifest both claim ownership of spec.replicas on the web and worker Deployments:

Error: UPGRADE FAILED: ... conflict occurred while applying object openstudio-server/worker apps/v1, Kind=Deployment:
Apply failed with 1 conflict: conflict with "kube-controller-manager" with subresource "scale" using apps/v1: .spec.replicas

Root cause

The chart's Deployment templates (templates/web/web-deploy.yaml, templates/worker/worker-deploy.yaml) declare a literal replicas: field, and separate HPA objects (web HPA, worker-hpa) also manage that same field via the /scale subresource. Server-side apply tracks field managers per-field; once the HPA controller has taken ownership of .spec.replicas (which happens as soon as it scales the Deployment even once), Helm's next apply of the same field from the chart's own rendered manifest is treated as a genuine conflict rather than a benign re-declaration of the initial/desired replica count.

Impact

Any helm upgrade against a running release with an active HPA fails outright unless the operator already knows to pass --force-conflicts (not documented anywhere in this repo). This will affect essentially every routine upgrade in production once the HPAs have scaled at least once, which is very likely by the time an upgrade is performed.

Reproduced

helm upgrade openstudio-server ./openstudio-server -f openstack/values-openstack.yaml -n openstudio-server
# fails with the conflict above

helm upgrade openstudio-server ./openstudio-server -f openstack/values-openstack.yaml -n openstudio-server --force-conflicts
# succeeds

Suggested fix (pick one or combine)

  1. Document --force-conflicts as a required flag for helm upgrade in README.md / deploy runbooks, alongside a short explanation of why (HPA + server-side apply field ownership).
  2. Better: stop declaring a hardcoded replicas: in the web/worker Deployment templates once an HPA is enabled for that workload — e.g. only render replicas: when the corresponding HPA is disabled, similar to patterns used elsewhere in the chart for optional fields. This avoids the conflict at the source rather than papering over it with --force-conflicts, and matches standard Helm/HPA guidance (Deployments under HPA control conventionally omit a static replicas after the first apply, or use Values.<workload>.replicas | default only when HPA is off).
  3. If (2) isn't feasible without bigger changes, at minimum wrap helm upgrade in any install/upgrade scripts (scripts/install-dry-run.sh or similar) with --force-conflicts so operators/CI don't hit this by surprise.

Related

Discovered while validating PR #106 (local-image-registry) end-to-end on a live OpenStack/Azimuth cluster; unrelated to the local-registry feature itself but blocked iterating on fixes for it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with templates/web/web-deploy.yaml and templates/worker/worker-deploy.yaml, then inspect the corresponding HPA templates and reproduce the documented helm upgrade against a release with an active HPA. Determine which proposed approach fits the chart, and verify that the upgrade path no longer fails on spec.replicas or that the required flag is documented and applied by the relevant runbook or script.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.