Prebuilds filling provisioner queues and blocking new jobs
- Lingua principale
- Nessun dato sulla lingua
- Stelle
- 3
- Fork
- 0
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Problem
Publishing a buggy template version can cause build jobs to never exit. When prebuilds are enabled, the bad version is enqueued across all available (in this case, in‑memory) provisioners, saturating their job queues. Because provisioners are full of stuck prebuild build jobs, new template versions cannot be published, leaving the deployment effectively deadlocked.
`coder prebuilds pause` correctly stops the reconciliation loop so new prebuild jobs aren’t picked up, but it does not clear in‑flight or queued jobs, so provisioners remain blocked. Prioritization of human jobs helps avoid user states being permanently blocked, but it doesn’t terminate the existing zombie builds. Customers need a one‑shot escape hatch to clear provisioners and recover to an operational state.
### Current behavior
- Admin publishes template version `vbad` with a bug.
- Prebuild controller enqueues builds for `vbad` across provisioners.
- Builds hang → provisioner job queue fill with stuck builds.
- Admin publishes `vfix` (corrected version), but publish/build jobs cannot schedule because queues are full of `vbad` builds.
- `coder prebuilds pause` prevents new prebuilds but doesn’t clear stuck jobs; manual per‑job cancel is required.
### Expected behavior
A safe, scoped bulk‑cleanup mechanism to quickly:
- Cancel in‑flight and queued prebuild build jobs for a selected template / template version.
- Optionally drain/flush affected provisioners to free capacity for a corrected publish/build.
## Impact
- Customer deployments can enter a deadlocked state where no new builds or publishes can be processed.
- Requires manual and potentially error-prone CLI work to enumerate and cancel each in‑progress job.
- Platform downtime while waiting for manual cleanup.
### Ideas
Some options to consider:
- Promoting a new active template version immediately after promoting a bad template version could stop in-progress build jobs on the previous template version.
- If you've just promoted a second/previous version, it's likely a fix.
- Also helps prevent users from creating a template on a previous version, though may stop their builds if not filtered for prebuilds.
- Could automagically clean up the problems.
- Runs the risk of weird sensible defaults for timing. Requires a
- Provide an in-dashboard option to clear prebuild jobs from provisioners in bulk.
- Should clear queues and in-progress jobs.
- Could be co-located with the provisioner jobs page could help clear things up
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Inizia tracciando il comando prebuilds pause, il prebuild controller e i provisioner jobs menzionati nell’issue. Definisci l’ambito per l’annullamento dei prebuild jobs accodati e in corso per template o versione, quindi determina se il draining del provisioner e un’azione nel dashboard fanno parte del design accettato; il lavoro è completato quando i publishes corretti possono essere pianificati senza la pulizia manuale di ogni job.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Ambito
- backend, cli, infrastructure
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100