tilt-dev / tilt-dev/tilt

registering images with k3d cluster registry host causes "failed to pull and unpack image"

Open
#5,328 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
10.1k
Forks
413
Avg merge
1d 10h
Merged PRs (30d)
6

Description

Steps to Reproduce

Create a k3d cluster with a registry

Note that when pushing to a k3d cluster, the registry host on the local network and on the cluster container runtime are different. They typically look like:

$ kubectl get configmap local-registry-hosting -n kube-public -o yaml
apiVersion: v1
data:
  localRegistryHosting.v1: |
    host: 127.0.0.1:35489
    hostFromClusterNetwork: ctlptl-k3d-registry:5000
    hostFromContainerRuntime: ctlptl-k3d-registry:5000
    help: https://k3d.io/usage/guides/registries/#using-a-local-registry

Hard-code the HostFromContainerRuntime in your tiltfile like this:

docker_build('ctlptl-k3d-registry:5000/my-img', ...)
Expected result

From the local network, tilt should push your image to 127.0.0.1:35489/my-img (but this is not necessarily the desired result, see "Notes")

Actual result

From the local network, tilt will push your image to 127.0.0.1:35489/ctlptl-k3d-registry_5000/my-img

Notes

There's a few different competing problems:

  1. Tilt REALLY doesn't expect you to be hard-coding local registry URLs in your Tiltfile (since they're supposed to be ephemeral and specific to a host machine/cluster).
  2. The image distribution spec is a mess, and makes poor guarantees about what's a valid image URL
  3. Tilt makes a best effort to rebase images for different hosts, but has some weird constraints on it due to the problems with the spec

In theory, Tilt COULD be smarter about figuring out what you're trying to do here, and push to the right URL. We'd have to special-case the scenario where the host in your tiltfile matches the host advertised by the cluster.

It also might make sense for Tilt to treat this as a hard error and/or warning. If you're hard-coding the k3d registry url in your tiltfile, you're probably deeply confused about registry URLs in general. Maybe it's Tilt's job to push you on the right path. By trying to fix things for you, we're just setting you up for more pain down the line.

I feel like this is an old Postel's Law dilemma (cf https://www.joelonsoftware.com/2008/03/17/martian-headsets/) but w/r/t to the image registry specs rather than web specs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Tiltfile docker_build example and the Kubernetes local-registry-hosting ConfigMap shown in the report, then trace how Tilt rebases the registry host for local pushes. The issue presents several possible outcomes—special handling, a warning, or an error—so completion requires an agreed behavior for hard-coded k3d registry hosts and a test covering the reported image path.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.