tilt-dev / tilt-dev/tilt

`local_resource` with pass-then-fail readiness_probe never shows as red

Open
#6,478 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
10.1k
Forks
413
Avg merge
1d 10h
Merged PRs (30d)
6

Description

Expected Behavior

When a readiness_probe on any resource (including local_resource) succeeds and turns green, then subsequently fails, the resource should show as unhealthy.

Current Behavior

Local resources specifically only show as red if the serve_cmd exits with any code. If the health check passes, then fails, the resource goes back to a "pending" state. The same is true for other types of probes as well.

The serve_cmd behavior is documented, but the readiness_probe behavior doesn't align with other uses of probes

Steps to Reproduce
  1. Create a local resource:
 local_resource(
   'never unhealthy',
   serve_cmd='while true; do sleep 1; done',
   readiness_probe=probe(
       exec=exec_action(
           command=['bash', '-c', 'exit $(cat counter)']
       ),
       period_secs=1,
       timeout_secs=1,
       success_threshold=1,
       failure_threshold=1,
   ),
   allow_parallel=True, # allow other resources to run in parallel
 )
  1. echo 0 > counter to make the readiness_probe pass
  2. echo 1 > counter to make it fail
  3. Note resource shows as pending in tilt UI
Context
tilt doctor Output
$ tilt doctor
Tilt: v0.33.21, built 2024-11-08
System: darwin-arm64
---
Docker
- Host: unix:///Users/xxxxxx/.docker/run/docker.sock
- Server Version: 27.3.1
- API Version: 1.47
- Builder: 2
- Compose Version: v2.30.3-desktop.1
---
Kubernetes
- Env: eks
- Context: dev-xxxx-a
- Cluster Name: arn:aws:eks:xxxxxx:xxxxxx:cluster/xxxxxxxx-xxxxxxxx
- Namespace: ingress-nginx
- Container Runtime: containerd
- Version: v1.29.10-eks-7f9249a
- Cluster Local Registry: none
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an `analytics_settings` call in your Tiltfile)
- User Mode: default
- Machine: 14c4502a17acda75b6e966fc7e1eca94
- Repo: 33EZAauGiC3WmaOoA66S7w==
...
About Your Use Case

We use local_resources to coordinate launching a number of local processes for local development (we don't use tilt for the k8s side of things at the moment). We utilize depends_on for ordering, so it becoming healthy is important. One of the resources we run is a watcher for files that will re-compile just that file using tsc rather than restarting the whole tilt resource. When tsc fails to compile, the output shows it, but the resource shows as healthy. We're using tsc-watch to drop a file when successful and delete it when failing, then setting the health check to fail if the file is not present so developers see a notice that it's failing.

We see the same functionality with running postgres locally with a similar tiltfile health check (so as not to re-trigger tiltfile re-evaluation).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how local_resource readiness_probe results are translated into resource status, using the reproduction steps to compare a passing probe followed by a failing probe. Done means a resource that turns green and then fails is shown as unhealthy or red, consistently with other probe users.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devtools
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.