apple / apple/container

[Bug]: `container k8s create --node-image` boots the requested image but writes the default image's version into the kubeadm config

Open
#2,246 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
49.9k
Forks
1.8k
Avg merge
1d 20h
Merged PRs (30d)
22

Description

### I have done the following
- [x] I have searched the existing issues
- [ ] If possible, I've reproduced the issue using the 'main' branch of this project

### Steps to reproduce

The defect is in the source, so the shortest reproduction is to hand kubeadm inside a 1.34 node image the config the plugin generates for it (`initConfigYAML` in `Sources/ContainerK8s/Support/K8sHelper+Bootstrap.swift`, whose `kubernetesVersion` comes out as `v1.35.5` for any `--node-image`):

```sh
container run --rm --entrypoint /bin/bash docker.io/kindest/node:v1.34.11 -c '
cat > /tmp/kubeadm-config.yaml <&1 | grep KubernetesVersion'
```

Output:

```
v1.34.11
registry.k8s.io/kube-apiserver:v1.35.5
registry.k8s.io/kube-controller-manager:v1.35.5
registry.k8s.io/kube-scheduler:v1.35.5
registry.k8s.io/kube-proxy:v1.35.5
registry.k8s.io/coredns/coredns:v1.12.1
registry.k8s.io/pause:3.10.1
registry.k8s.io/etcd:3.6.5-0
[WARNING KubernetesVersion]: Kubernetes version is greater than kubeadm version. Please consider to upgrade kubeadm. Kubernetes version: 1.35.5. Kubeadm version: 1.34.x
```

I could not run `container k8s create --node-image docker.io/kindest/node:v1.34.11` to the end on the shipped 1.3.1 because node prep stops earlier (#2120), so the end state below is expected rather than observed.

### Problem description

`kubernetesVersion()` in `K8sHelper+Bootstrap.swift` takes no argument: it parses the tag out of the static `K8sHelper.nodeImage` constant (`docker.io/kindest/node:v1.35.5@sha256:...`). The `--node-image` value flows to `LinuxNodeProvisioner`, which pulls and boots it, but never to `bootstrapControlPlane`. So every cluster's kubeadm config says `kubernetesVersion: v1.35.5`.

kubeadm treats a newer control-plane version as a warning, and the image list above is what it would pull, so a user who asks for a 1.34 node would be expected to get a 1.35.5 control plane on a 1.34 kubelet. For older images the gap grows until it exceeds the documented n-3 skew (kubelet may be up to three minor versions older than kube-apiserver): a v1.32 node lands exactly at that limit, v1.31 one minor past it. Below 1.31 kubeadm rejects the `v1beta4` config outright (`experimental API spec: "kubeadm.k8s.io/v1beta4" is not allowed`, checked in `kindest/node:v1.30.8`; `v1.31.14` accepts it), so 1.31 is the effective floor for `--node-image` regardless of this bug.

Expected: the kubeadm config carries the version of the image actually booted. The simplest fix is to omit `kubernetesVersion` entirely: kubeadm then resolves it itself, to the latest patch of its own minor when it can reach dl.k8s.io and to its own exact version when it cannot (both checked inside `kindest/node:v1.34.0`, which resolved to v1.34.11 online and to v1.34.0 with the fetch blocked, warning only). Parsing the resolved image tag also works but `kubeadm config validate` hard-errors on a non-semver custom tag, so it needs a fallback; asking the node with `kubeadm version -o short` is a third option. Happy to attempt a PR if you would prefer, though I would be relying on CI to compile it (Command Line Tools only here).

Related: PR #2244 documents `--node-image` for running several Kubernetes versions side by side, which is what surfaced this.

### Environment
- OS: macOS 26.2 (25C56), Apple silicon
- Xcode: Command Line Tools only
- Container: container CLI version 1.3.1 (Homebrew); `Sources/ContainerK8s` read at `main`

### Code of Conduct
- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start in Sources/ContainerK8s/Support/K8sHelper+Bootstrap.swift, tracing initConfigYAML and kubernetesVersion(), then follow --node-image through LinuxNodeProvisioner to bootstrapControlPlane. Run the provided kubeadm reproduction with a v1.34 node image. Done means the generated kubeadm configuration uses the booted image's Kubernetes version rather than the default, while preserving supported custom-image behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, swift
Domain
cli, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.