Detect when pods are stuck in an Unschedulable state
@milas is already working on this.
Since Mar 30, 2021.
- Dominant language
- Go
- Stars
- 10.1k
- Forks
- 413
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 6
Description
Expected Behavior
- If a pod is never going to be scheduled, show the resource as having an error and fail
tilt ci
Current Behavior
- Resource will show as pending forever in
tilt upand hangtilt ciindefinitely
Steps to Reproduce
- Taint your node, e.g. for kind:
kubectl taint nodes kind-control-plane key=value:NoSchedule --overwrite - Run
tilt up(ortilt ci) for aTiltfilethat deploys a pod - Observe that pod is stuck in pending forever (or
tilt cihangs)
Extra Context
It's worth noting that a first attempt at addressing this via #4386 had to be reverted because it's not atypical for pods to end up in Unschedulable transiently for different reasons. That said, it'd be helpful to have better heuristics for when the situation isn't expected to imminently resolve and reflect that more usefully as an "error" within Tilt.
About Your Use Case
With tilt up, this is more of a nuisance - we DO show the unschedulable event at least but don't show it as an error (so you might not notice for a while)
For tilt ci, it means things hang indefinitely
See also #437 - there used to be some logic around this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.