coder / coder/internal

Prebuilds filling provisioner queues and blocking new jobs

Aperta
#924 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Nessun dato sulla lingua
Stelle
3
Fork
0
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Problem

Publishing a buggy template version can cause build jobs to never exit. When prebuilds are enabled, the bad version is enqueued across all available (in this case, in‑memory) provisioners, saturating their job queues. Because provisioners are full of stuck prebuild build jobs, new template versions cannot be published, leaving the deployment effectively deadlocked.

`coder prebuilds pause` correctly stops the reconciliation loop so new prebuild jobs aren’t picked up, but it does not clear in‑flight or queued jobs, so provisioners remain blocked. Prioritization of human jobs helps avoid user states being permanently blocked, but it doesn’t terminate the existing zombie builds. Customers need a one‑shot escape hatch to clear provisioners and recover to an operational state.

### Current behavior

- Admin publishes template version `vbad` with a bug.
- Prebuild controller enqueues builds for `vbad` across provisioners.
- Builds hang → provisioner job queue fill with stuck builds.
- Admin publishes `vfix` (corrected version), but publish/build jobs cannot schedule because queues are full of `vbad` builds.
- `coder prebuilds pause` prevents new prebuilds but doesn’t clear stuck jobs; manual per‑job cancel is required.

### Expected behavior

A safe, scoped bulk‑cleanup mechanism to quickly:
- Cancel in‑flight and queued prebuild build jobs for a selected template / template version.
- Optionally drain/flush affected provisioners to free capacity for a corrected publish/build.

## Impact
- Customer deployments can enter a deadlocked state where no new builds or publishes can be processed.
- Requires manual and potentially error-prone CLI work to enumerate and cancel each in‑progress job.
- Platform downtime while waiting for manual cleanup.

### Ideas

Some options to consider:
- Promoting a new active template version immediately after promoting a bad template version could stop in-progress build jobs on the previous template version.
- If you've just promoted a second/previous version, it's likely a fix.
- Also helps prevent users from creating a template on a previous version, though may stop their builds if not filtered for prebuilds.
- Could automagically clean up the problems.
- Runs the risk of weird sensible defaults for timing. Requires a
- Provide an in-dashboard option to clear prebuild jobs from provisioners in bulk.
- Should clear queues and in-progress jobs.
- Could be co-located with the provisioner jobs page could help clear things up

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia tracciando il comando prebuilds pause, il prebuild controller e i provisioner jobs menzionati nell’issue. Definisci l’ambito per l’annullamento dei prebuild jobs accodati e in corso per template o versione, quindi determina se il draining del provisioner e un’azione nel dashboard fanno parte del design accettato; il lavoro è completato quando i publishes corretti possono essere pianificati senza la pulizia manuale di ogni job.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Ambito
backend, cli, infrastructure
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.