kthcloud / kthcloud/go-deploy

Deployments with failed mounts dont get deleted

Open
#688 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
3
Forks
1
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
Deployments that experience a mount failure cannot be deleted. The deletion process appears to hang or fail due to issues with removing the associated PersistentVolumeClaim (PVC). It seems like there is a create storage job that still tries to create it everytime.

---

**To Reproduce**
Steps to reproduce the behavior:

1. Create a deployment with a volume mount that will fail (e.g., invalid NFS path or storage configuration).
2. Wait for the deployment to enter a failed or crash-loop state due to the mount issue.
3. Attempt to delete the deployment.
4. Observe that the deletion process fails or times out.

---

**Expected behavior**
Deployments should be deleted even if the PVC is missing, invalid, or cannot be mounted. The deletion logic should either skip over non-existent PVCs or handle stuck PVCs gracefully without blocking the overall deployment deletion.

---

**Screenshots**
N/A (the issue is visible in logs).

---

**Additional context**
This issue likely occurs because the deletion job waits indefinitely for a PVC that is either:

* already deleted,
* stuck in a terminating state, or
* never created successfully due to an earlier mount error.

Log excerpt:

```json
{"level":"info","ts":1760083581.4472415,"logger":"default","msg":"failed job 069fe4da-3436-42c6-b991-b41432f8f545 (deleteDeployment), attempt: 1/5 delay: 30s \n\tdue to: failed to delete deployment. \n\tdue to: failed to delete k8s for deployment 05fd7745-860b-4c23-a1a2-c1a183943a1d. \n\tdue to: failed to delete k8s pvc root-philipzi-trino-coordinator-1. \n\tdue to: timeout waiting for pvc root-philipzi-trino-coordinator-1 to be deleted"}
```

Potential fix:
Add a safeguard in the deletion logic to detect non-existent or stuck PVCs / cancel all jobs related to it and proceed with deployment deletion without timing out.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the deleteDeployment job and the Kubernetes deletion path described in the logs, focusing on PVC removal and its timeout. Reproduce the failed-mount scenario, then verify that deployment deletion proceeds when the PVC is missing, stuck terminating, or was never created, without waiting indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.