kubernetes / kubernetes/release

kube-cross: move off Debian bullseye (EOL 2026-08-31)

Open
#4,522 5 comments 0 reactions 1 assignee Claimed by @cpanato View on GitHub
area/release-eng kind/feature needs-priority sig/release
Dominant language
Go
Stars
504
Forks
556
Avg merge
18h 43m
Merged PRs (30d)
33

Description

#### What would you like to be added:

Migrate the `kube-cross` image off Debian 11 (bullseye) onto a supported base.

All five `kube-cross` variants currently build on `debian:bullseye-slim`:

```
v1.38-go1.27-bullseye
v1.37-go1.26-bullseye
v1.36-go1.26-bullseye
v1.35-go1.26-bullseye
v1.35-go1.25-bullseye
```

#### Why is this needed:

Debian 11 reached the end of its LTS window on **2026-08-31**, and `kube-cross`
builds broke the following week. The `bullseye-security` `InRelease` file is
signed `Mon, 31 Aug 2026 21:13:04 UTC` with `Valid-Until: Mon, 07 Sep 2026
21:13:04 UTC`, and it will not be re-signed, so `apt-get update` fails the whole
transaction:

```
E: Release file for http://deb.debian.org/debian-security/dists/bullseye-security/InRelease
is expired (invalid since 12h 8min 45s). Updates for this repository will not be applied.
```

Every variant of `pull-release-image-kube-cross` fails on this.

Setting `Acquire::Check-Valid-Until "false"` is *not* sufficient on its own: the
security pool is being pruned behind the CDN, so `deb.debian.org` intermittently
returns 404 for packages the index still advertises (reproduced locally — 2 of 3
builds failed fetching `openssl_1.1.1w-0+deb11u8_amd64.deb`, with the same URL
returning 200 and 404 minutes apart from the same Fastly node).

`bullseye` is not on `archive.debian.org` yet — only `buster` and older are there
for `debian-security` — so repointing at the archive is not an option today.

##### Current stopgap

kubernetes/release#4521 pins bullseye's apt sources to
`snapshot.debian.org/archive/.../20260824T000000Z` (the snapshot the base image
was built from; `debian:bullseye-slim` ships those entries in `sources.list`,
commented out). That is an immutable archive which still carries the security
suite, so the package set is unchanged and builds are reproducible again.

It is only a stopgap. It leaves `kube-cross` frozen on an EOL base that will
receive no further security updates, and by extension every Kubernetes binary
built with it.

##### What needs to happen

Moving the base is more involved than a codename bump, which is why it is filed
separately:

1. **Pick the target base.** `bookworm` (glibc 2.36) or `trixie` (glibc 2.41),
versus bullseye's current glibc 2.31.
2. **The glibc floor is user-facing.** `kube-cross` produces the officially
released Kubernetes binaries, so raising it raises the minimum glibc required
to run `kubelet`/`kubectl` on the target host. This likely needs a heads-up to
sig-release and a release note, and it interacts with #3246.
3. **The codename is baked into image tags**
(`v1.38.0-go1.27.1-bullseye.0`), and kubernetes/kubernetes pins those:
- `master` and `release-1.37` → `v1.37.0-go1.26.5-bullseye.0`
- `release-1.36` → `v1.36.0-go1.26.5-bullseye.0`
- `release-1.35` → `v1.35.0-go1.26.5-bullseye.0`

in `build/build-image/cross/VERSION`, so the rollout needs coordinating across
release branches.
4. **`k8s-cloud-builder` inherits the problem.** It is
`FROM registry.k8s.io/build-image/kube-cross:${KUBE_CROSS_VERSION}` and runs
`apt-get update` itself, and all four of its variants pin already-published
bullseye `kube-cross` images that predate the snapshot fix. It only recovers
once repointed at newly published images.
5. **Decide how far back to go.** The older variants exist to keep building
supported release branches; a decision is needed on whether they move too or
stay pinned to the snapshot until they age out.

For reference, kubernetes/release#4302 already did this move for
`k8s-ci-builder` (bullseye → bookworm). `go-runner` and `releng-ci` are already
on bookworm/trixie, so `kube-cross` (and `k8s-cloud-builder` behind it) is the
last image on bullseye.

/sig release
/area release-eng

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.