Handle Deployment-to-StatefulSet transitions when adding a persistent volume
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Describe the bug
First-class Kubernetes persistent-volume support automatically promotes a workload from a `Deployment` to a `StatefulSet` when a persistent-volume binding is added. When that change is applied to an existing Helm release, the previously deployed `Deployment` can remain alongside the newly created `StatefulSet`.
Both workloads retain the same application labels, so the existing `Service` selects pods from both controllers. The old Deployment pods do not have the new persistent-volume mount, while the StatefulSet pods do. Traffic can therefore be sent to workloads with different filesystem and state semantics.
This is a general lifecycle/design problem in the Kubernetes deployment support, not a cluster-specific issue. Changing the generated Kubernetes kind changes the identity and shape of the resource, so this transition needs explicit handling or clear guardrails.
### Expected Behavior
After adding a persistent volume and redeploying an existing workload, there should be exactly one active workload shape behind its Service. Users should not unexpectedly end up with both the old Deployment and the replacement StatefulSet serving traffic.
The deployment experience should either:
- safely migrate the Helm release from Deployment to StatefulSet;
- detect the existing workload shape and provide an actionable migration step; or
- prevent the upgrade with a clear explanation when it cannot be performed safely.
Any solution should consider release ownership, readiness, downtime, rollback behavior, and avoiding deletion of user-managed resources.
### Steps To Reproduce
1. Define a project/container resource in an Aspire AppHost targeting Kubernetes or AKS.
2. Deploy it without a persistent-volume binding. The generated workload is a Deployment.
3. Add a `KubernetesPersistentVolumeResource` and bind it to the same workload with `WithPersistentVolume(...)`.
4. Deploy again to the same Helm release.
5. Observe that a StatefulSet is created while the previous Deployment remains.
6. Observe that the Service selector includes pods from both controllers, although only the StatefulSet pod has the PVC mounted.
In the observed case, the Service endpoints contained one Deployment pod and one StatefulSet pod. Requests sent to the Deployment pod failed because `/data` did not exist; requests sent to the StatefulSet pod reached the mounted volume.
### Exceptions (if any)
The old Deployment pod failed application writes with:
```text
System.IO.DirectoryNotFoundException: Could not find a part of the path '/data/guids.txt'.
```
A separate volume-ownership issue caused the StatefulSet pod to report `UnauthorizedAccessException`; that permission behavior is not the core concern tracked here.
### Aspire doctor output
N/A
### Anything else?
This is focused follow-up work from #16999 and the persistent-volume implementation in #16929. The deferred-work section of #16999 already calls out the in-place kind-change problem and suggests researching a Helm `pre-upgrade` hook that removes a prior Deployment before installing the StatefulSet.
More research is required before selecting that design. In particular:
- Confirm Helm's behavior when a release manifest changes from Deployment to StatefulSet with the same logical workload name.
- Determine why the old Deployment remains in the observed upgrade path.
- Evaluate whether a pre-upgrade hook, explicit migration step, generated cleanup resource, or a different publishing strategy is safest.
- Ensure failed upgrades and Helm rollbacks do not leave the application without a workload or restore two competing workloads.
- Only remove resources demonstrably owned by the same Helm release.
- Ensure the old workload stops matching the Service before mixed-shape traffic can occur.
This transition should be streamlined because adding persistence is a natural evolution of an application and users may not realize it changes the generated Kubernetes workload kind.
Contributor guide
Research direction
Start with the reproduction in the issue and inspect Helm behavior when the generated workload changes from Deployment to StatefulSet, using the deferred-work notes in #16999 and the persistent-volume implementation from #16929. Compare ownership, upgrade, rollback, and Service-selector behavior for the proposed migration paths. Done means an existing release has one active workload shape, avoids mixed traffic, and does not remove resources outside the Helm release.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100