tilt-dev / tilt-dev/tilt

triggering a live_update defined on a k8s base image leads to the resource pending indefinitely

Open
#4,539 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
10.1k
Forks
413
Avg merge
1d 10h
Merged PRs (30d)
6

Description

Steps to Reproduce

Setup:

Define a base image and a child image, with separate build contexts. Define a k8s resource that runs that image. Give the base image a live_update.

OR check out this branch and go to this directory: https://github.com/tilt-dev/tilt/tree/matt/live_update_on_base_image/integration/live_update_base_image

Run tilt, touch a file in the base image's context.

Expected Behavior

I'm not entirely sure what the expected behavior should be, but the behavior used to be that the base image's live_update was silently ignored and we did an image build

silently ignoring the live_update seems bad, but is better than just making the resource irrecoverably pending
some (not mutually exclusive) options:

  1. try to restore the behavior of only honoring live_update on the leaf image
  2. generate a warning when doing a full build but there's a matching live_update on an ancestor image
  3. generate a warning/error if an image has a live_update but is not a leaf image of any resource

Given that it seems this bug has existed for 6 months and no one has noticed, it's probably not worth sinking a bunch of time into, and I was rather filing this because:

  1. we often get confused about where to define live_update when base images are installed
  2. if a user reports this or a related issue in the future, maybe this will help
Current Behavior

When you touch the file, Tilt shows the k8s resource as "pending", but does not output anything.

Context
  1. One can work around this by defining the live_updates on the child images that are actually referenced in the k8s images (e.g., this integration test, so afaik this is a usability issue and not a functionality issue - the main potential impact is that a user wants to live_update a base image, tries to do it a perfectly logical way, and gets stuck on Tilt misbehaving.
  2. I'm mildly surprised I haven't seen any reports of this, and wonder if there's some other key element to my repro and maybe the problem isn't as general as I thought.
  3. Whatever the behavior, it'd probably be good to have an integration test for this, given how easy it is to get mixed up on the proper way to do live_updates with base images.
  4. tilt get session -ojson says "waitReason": "waiting-for-deploy"
  5. It looks like this is because in IsLiveUpdateTargetWaitingOnDeploy, RunningContainersForTargetForOnePod only looks for containers running the iTarget's ref directly, and not containers running images descended from the iTarget. I've verified this behavior was introduced in 9c39361a8. I haven't put much thought into whether there's a simple fix for this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with IsLiveUpdateTargetWaitingOnDeploy and RunningContainersForTargetForOnePod, which the report identifies as the suspected path, then reproduce the case from integration/live_update_base_image and its Tiltfile. Decide and document the intended base-image behavior, add coverage for touching a base-image file, and verify the resource no longer remains pending indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.