Prebuilds filling provisioner queues and blocking new jobs
- Dominant language
- No language data
- Stars
- 3
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Publishing a buggy template version can cause build jobs to never exit. When prebuilds are enabled, the bad version is enqueued across all available (in this case, in‑memory) provisioners, saturating their job queues. Because provisioners are full of stuck prebuild build jobs, new template versions cannot be published, leaving the deployment effectively deadlocked.
`coder prebuilds pause` correctly stops the reconciliation loop so new prebuild jobs aren’t picked up, but it does not clear in‑flight or queued jobs, so provisioners remain blocked. Prioritization of human jobs helps avoid user states being permanently blocked, but it doesn’t terminate the existing zombie builds. Customers need a one‑shot escape hatch to clear provisioners and recover to an operational state.
### Current behavior
- Admin publishes template version `vbad` with a bug.
- Prebuild controller enqueues builds for `vbad` across provisioners.
- Builds hang → provisioner job queue fill with stuck builds.
- Admin publishes `vfix` (corrected version), but publish/build jobs cannot schedule because queues are full of `vbad` builds.
- `coder prebuilds pause` prevents new prebuilds but doesn’t clear stuck jobs; manual per‑job cancel is required.
### Expected behavior
A safe, scoped bulk‑cleanup mechanism to quickly:
- Cancel in‑flight and queued prebuild build jobs for a selected template / template version.
- Optionally drain/flush affected provisioners to free capacity for a corrected publish/build.
## Impact
- Customer deployments can enter a deadlocked state where no new builds or publishes can be processed.
- Requires manual and potentially error-prone CLI work to enumerate and cancel each in‑progress job.
- Platform downtime while waiting for manual cleanup.
### Ideas
Some options to consider:
- Promoting a new active template version immediately after promoting a bad template version could stop in-progress build jobs on the previous template version.
- If you've just promoted a second/previous version, it's likely a fix.
- Also helps prevent users from creating a template on a previous version, though may stop their builds if not filtered for prebuilds.
- Could automagically clean up the problems.
- Runs the risk of weird sensible defaults for timing. Requires a
- Provide an in-dashboard option to clear prebuild jobs from provisioners in bulk.
- Should clear queues and in-progress jobs.
- Could be co-located with the provisioner jobs page could help clear things up
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.