apple / apple/container

[Request]: support KubeSolo for the k8s plugin

Open
#2,225 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
49.9k
Forks
1.8k
Avg merge
1d 20h
Merged PRs (30d)
22

Description

### Feature or enhancement request details

## Summary

Hello! I'm a core maintainer of [KubeSolo](https://github.com/portainer/kubesolo) and would like to contribute support for running it as a `container k8s` node, via `--node-image` pointing at a KubeSolo image.

KubeSolo already runs on Apple Containers through plain `container run`, so this should be a relatively small task. The one blocker is that the bootstrap flow assumes kubeadm, and KubeSolo bootstraps itself.

## Background

KubeSolo is a single-node Kubernetes distribution that runs the API server, controller-manager, kubelet, kube-proxy and containerd in one process.
It has no systemd and does not use kubeadm. It generates its own PKI and bootstraps its control plane directly from its entrypoint.

## Where Apple Containers support stands today

KubeSolo already runs on Apple Containers, and we [document it](https://github.com/portainer/kubesolo/blob/develop/docs/configuration/container-mode.md#running-with-apple-containers):

```bash
container run -d --name kubesolo --cap-add ALL \
--tmpfs /tmp --tmpfs /run \
-v kubesolo-data:/var/lib/kubesolo \
-p 16443:6443 \
portainer/kubesolo:v1.2.0
```

That works, but it is too manual. Once a cluster is bootstrapped, users then have to extract the kubeconfig with `container exec` and hand-edit the server address, and there is no lifecycle management. Instead, we would much rather users simply run:

```bash
container k8s create --node-image portainer/kubesolo:v1.2.0
```

That fits the native workflow you are already building, and would give KubeSolo users on macOS the same one-command experience they get on other engines. Commands like `k8s list`, `start`, `delete` and `write-config` will also be gold!

## What I tried

```bash
container k8s create --node-image kubesolo:k8s-plugin --name kubesolo-dev
```

To satisfy the plugin's bootstrap script I added three stubs to a KubeSolo image:

| Stub | Purpose |
|---|---|
| `/usr/bin/kubeadm` | KubeSolo has already bootstrapped by the time this runs; the stub publishes its existing kubeconfig to `/etc/kubernetes/admin.conf` |
| `/usr/bin/systemctl` | no-op — KubeSolo supervises its own containerd and does not read `/etc/containerd/conf.d` |
| `kubectl` wrapper | drops the kindnet apply — KubeSolo ships its own CNI, and two would conflict |

Result:

```
Waiting for node to boot: ["node": kubesolo-dev]
Preparing node: ["id": kubesolo-dev]
Running kubeadm init: ["node": kubesolo-dev]
Removing control-plane taint for single-node scheduling: ["node": kubesolo-dev]
Removing kubeconfig: ["cluster": kubesolo-dev, "path": /Users/steven/.kube/config]
Error: failed to start process ... in container kubesolo-dev
(cause: "vmexec error: failed to find target executable /bin/kubectl")
```

The node booted, bootstrap completed, the kubeconfig was published, and the flow reached post-bootstrap steps.

## Findings

**1. The containerd readiness check needed no changes.**
KubeSolo already symlinks its socket to `/run/containerd/containerd.sock`, so the plugin's `test -S` passed as-is. Worth noting since it suggests that part of the contract is already generic.

**2. The post-bootstrap steps are not needed either.**
After `kubeadm init` the plugin applies kindnet and removes the control-plane taint, both via `kubectl`.
KubeSolo ships its own CNI, so applying kindnet would leave two CNIs competing for pod networking.
It is also single-node by design and sets no control-plane taint, so there is nothing to remove.

**3. The bootstrap block is the only real coupling.**
Everything around it is distribution-agnostic — boot the node, wait for a runtime, read a kubeconfig back out.
Only the containerd config write, `systemctl restart`, `kubeadm init`, the kindnet apply and the taint removal assume kind specifically.

## Proposal

**Add a way for a node image to declare that it bootstraps itself.**

When set, the plugin would skip the whole bootstrap block:

| Step | Why it can be skipped |
|---|---|
| write `/etc/containerd/conf.d/…` | the distribution configures its own runtime |
| `systemctl restart containerd` | no systemd; the runtime is already supervised |
| `kubeadm init` | the control plane is already up |
| `kubectl apply -f kindnet.yaml` | the distribution provides its own CNI |
| remove control-plane taint | single-node distributions set none |

Instead, wait for a kubeconfig to appear at a declared path, defaulting to `/etc/kubernetes/admin.conf`.

The mechanism is entirely your call: something like a `--bootstrap none` flag, an image label, or probing for a marker file would all work for us. The requirement from our side is pretty much boot the image, wait, read the kubeconfig.

## What I can contribute

- A PR implementing the self-bootstrap hook, following the design you prefer
- Testing a branch, if you'd rather shape it yourselves
- The shim image and steps, if useful for reproduction

If this isn't a direction you want `container k8s` to take, that's completely fair! Knowing either way is valuable, and we'll document the `container run` path for our users instead.

Thanks for considering 😄

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Trace the `container k8s create --node-image` bootstrap flow, focusing on the kubeadm, containerd, systemctl, kindnet, taint-removal, and kubeconfig steps described in the issue. Run the KubeSolo reproduction command or equivalent node-image flow first. Done means a self-bootstrapping image can skip distribution-specific bootstrap work, wait for its declared kubeconfig path, and support the existing k8s lifecycle commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, swift
Domain
cli, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.