tilt-dev / tilt-dev/tilt

Detect when pods are stuck in an Unschedulable state

Open
#4,385 5 comments 1 reaction 1 assignee View on GitHub

@milas is already working on this.

Since Mar 30, 2021.

enhancement
Dominant language
Go
Stars
10.1k
Forks
413
Avg merge
1d 10h
Merged PRs (30d)
6

Description

Expected Behavior
  • If a pod is never going to be scheduled, show the resource as having an error and fail tilt ci
Current Behavior
  • Resource will show as pending forever in tilt up and hang tilt ci indefinitely
Steps to Reproduce
  1. Taint your node, e.g. for kind: kubectl taint nodes kind-control-plane key=value:NoSchedule --overwrite
  2. Run tilt up (or tilt ci) for a Tiltfile that deploys a pod
  3. Observe that pod is stuck in pending forever (or tilt ci hangs)
Extra Context

It's worth noting that a first attempt at addressing this via #4386 had to be reverted because it's not atypical for pods to end up in Unschedulable transiently for different reasons. That said, it'd be helpful to have better heuristics for when the situation isn't expected to imminently resolve and reflect that more usefully as an "error" within Tilt.

About Your Use Case

With tilt up, this is more of a nuisance - we DO show the unschedulable event at least but don't show it as an error (so you might not notice for a while)
For tilt ci, it means things hang indefinitely

See also #437 - there used to be some logic around this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.