triggering a live_update defined on a k8s base image leads to the resource pending indefinitely
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 10.1k
- Forks
- 413
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 6
Description
Steps to Reproduce
Setup:
Define a base image and a child image, with separate build contexts. Define a k8s resource that runs that image. Give the base image a live_update.
OR check out this branch and go to this directory: https://github.com/tilt-dev/tilt/tree/matt/live_update_on_base_image/integration/live_update_base_image
Run tilt, touch a file in the base image's context.
Expected Behavior
I'm not entirely sure what the expected behavior should be, but the behavior used to be that the base image's live_update was silently ignored and we did an image build
silently ignoring the live_update seems bad, but is better than just making the resource irrecoverably pending
some (not mutually exclusive) options:
- try to restore the behavior of only honoring live_update on the leaf image
- generate a warning when doing a full build but there's a matching live_update on an ancestor image
- generate a warning/error if an image has a live_update but is not a leaf image of any resource
Given that it seems this bug has existed for 6 months and no one has noticed, it's probably not worth sinking a bunch of time into, and I was rather filing this because:
- we often get confused about where to define live_update when base images are installed
- if a user reports this or a related issue in the future, maybe this will help
Current Behavior
When you touch the file, Tilt shows the k8s resource as "pending", but does not output anything.
Context
- One can work around this by defining the live_updates on the child images that are actually referenced in the k8s images (e.g., this integration test, so afaik this is a usability issue and not a functionality issue - the main potential impact is that a user wants to live_update a base image, tries to do it a perfectly logical way, and gets stuck on Tilt misbehaving.
- I'm mildly surprised I haven't seen any reports of this, and wonder if there's some other key element to my repro and maybe the problem isn't as general as I thought.
- Whatever the behavior, it'd probably be good to have an integration test for this, given how easy it is to get mixed up on the proper way to do live_updates with base images.
tilt get session -ojsonsays"waitReason": "waiting-for-deploy"- It looks like this is because in
IsLiveUpdateTargetWaitingOnDeploy,RunningContainersForTargetForOnePodonly looks for containers running the iTarget's ref directly, and not containers running images descended from the iTarget. I've verified this behavior was introduced in 9c39361a8. I haven't put much thought into whether there's a simple fix for this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with IsLiveUpdateTargetWaitingOnDeploy and RunningContainersForTargetForOnePod, which the report identifies as the suspected path, then reproduce the case from integration/live_update_base_image and its Tiltfile. Decide and document the intended base-image behavior, add coverage for touching a base-image file, and verify the resource no longer remains pending indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100