tilt-dev / tilt-dev/tilt-extensions

`podman_build` pushes a new tag on every build, even when the image is unchanged

Open
#697 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Starlark
Stars
243
Forks
194
Avg merge
3d 4h
Merged PRs (30d)
5

Description

podman_build and podman_build_with_restart both push $EXPECTED_REF. In normal mode Tilt sets that to tilt-build-<unix-timestamp>, so each build pushes a tag that nothing will ever reuse, even when podman rebuilt nothing and the image is byte-for-byte the same.

The registry garbage collector can't drop anything, because every old layer is still reachable from a tag. It only grows from there.

The two entry points don't get off equally lightly. podman_build_with_restart just leaves the tags behind. Plain podman_build deploys the timestamped ref as well, so the pods get rolled on every build.

Steps to reproduce

kind, with the local registry from https://kind.sigs.k8s.io/docs/user/local-registry/. Then three files.

Containerfile:

FROM docker.io/alpine:3.21
CMD ["sleep", "infinity"]

Tiltfile:

load('ext://podman', 'podman_build')

podman_build('localhost:5000/mre', '.', deps=['trigger'])
k8s_yaml('mre.yaml')

mre.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: podman-tag-mre
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mre
  namespace: podman-tag-mre
spec:
  replicas: 1
  selector:
    matchLabels: {app: mre}
  template:
    metadata:
      labels: {app: mre}
    spec:
      containers:
        - name: mre
          image: localhost:5000/mre

Run tilt up, then touch trigger three times a few seconds apart. That's four builds counting the initial one, and here is what the registry and the cluster hold afterwards:

$ curl -s http://localhost:5000/v2/mre/tags/list
{"name":"mre","tags":["tilt-build-1788467962","tilt-build-1788468004","tilt-build-1788468012","tilt-build-1788468020"]}

# Docker-Content-Digest of each of those tags
# tilt-build-1788467962  sha256:6aa8e6fbcccfa4fbcc202727409061c465322d443480d724044a878a22ce683c
# tilt-build-1788468004  sha256:6aa8e6fbcccfa4fbcc202727409061c465322d443480d724044a878a22ce683c
# tilt-build-1788468012  sha256:6aa8e6fbcccfa4fbcc202727409061c465322d443480d724044a878a22ce683c
# tilt-build-1788468020  sha256:6aa8e6fbcccfa4fbcc202727409061c465322d443480d724044a878a22ce683c

$ kubectl get rs -n podman-tag-mre -o custom-columns=RS:.metadata.name,REPLICAS:.spec.replicas,IMAGE:.spec.template.spec.containers[0].image
RS               REPLICAS   IMAGE
mre-59f88c7f57   1          localhost:5000/mre:tilt-build-1788468020
mre-6bf486c8c    0          localhost:5000/mre:tilt-build-1788468012
mre-8fbfb6566    0          localhost:5000/mre:tilt-build-1788467962
mre-dc6b66b9c    0          localhost:5000/mre:tilt-build-1788468004

Context

  • Tilt v0.37.6
  • podman 5.8.6, rootless, no local Docker involved
  • tilt-extensions at a8701c0
  • kind 1.35.5 with a local registry

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the supplied kind reproduction with the local registry, then inspect the podman_build and podman_build_with_restart entry points. Compare registry tags and Kubernetes ReplicaSets after repeated unchanged builds; done means unchanged images do not accumulate timestamped tags or roll pods unnecessarily.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
build-system, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.