Deployments with failed mounts dont get deleted
- Dominant language
- Go
- Stars
- 3
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Deployments that experience a mount failure cannot be deleted. The deletion process appears to hang or fail due to issues with removing the associated PersistentVolumeClaim (PVC). It seems like there is a create storage job that still tries to create it everytime.
---
**To Reproduce**
Steps to reproduce the behavior:
1. Create a deployment with a volume mount that will fail (e.g., invalid NFS path or storage configuration).
2. Wait for the deployment to enter a failed or crash-loop state due to the mount issue.
3. Attempt to delete the deployment.
4. Observe that the deletion process fails or times out.
---
**Expected behavior**
Deployments should be deleted even if the PVC is missing, invalid, or cannot be mounted. The deletion logic should either skip over non-existent PVCs or handle stuck PVCs gracefully without blocking the overall deployment deletion.
---
**Screenshots**
N/A (the issue is visible in logs).
---
**Additional context**
This issue likely occurs because the deletion job waits indefinitely for a PVC that is either:
* already deleted,
* stuck in a terminating state, or
* never created successfully due to an earlier mount error.
Log excerpt:
```json
{"level":"info","ts":1760083581.4472415,"logger":"default","msg":"failed job 069fe4da-3436-42c6-b991-b41432f8f545 (deleteDeployment), attempt: 1/5 delay: 30s \n\tdue to: failed to delete deployment. \n\tdue to: failed to delete k8s for deployment 05fd7745-860b-4c23-a1a2-c1a183943a1d. \n\tdue to: failed to delete k8s pvc root-philipzi-trino-coordinator-1. \n\tdue to: timeout waiting for pvc root-philipzi-trino-coordinator-1 to be deleted"}
```
Potential fix:
Add a safeguard in the deletion logic to detect non-existent or stuck PVCs / cancel all jobs related to it and proceed with deployment deletion without timing out.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the deleteDeployment job and the Kubernetes deletion path described in the logs, focusing on PVC removal and its timeout. Reproduce the failed-mount scenario, then verify that deployment deletion proceeds when the PVC is missing, stuck terminating, or was never created, without waiting indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100