bug(local-dev): local development setup fails on all non-Kind Kubernetes clusters (K3s, Minikube, K3d)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 365
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 84
Description
What happened?
The local development environment (run via make run/pipecd and the associated local Helm values) is tightly coupled with Kind. When attempting to start the local development cluster on any non-Kind distribution (such as K3s, Minikube, K3d, or MicroK8s), the setup fails due to registry image pulling and persistent storage permission blockers.
Steps to Reproduce
- Configure your Kubernetes context to point to a local K3s or Minikube cluster.
- Enable local storage by setting
localStorageVolumes.enabled: truein the Helm values. - Run
make run/pipecd. - Observe:
- The cluster fails to pull the newly built PipeCD image (
ErrImagePull) because it cannot reach/resolvelocalhost:5001. - If the image is manually sideloaded, the database (
mysql), cache (redis), and storage (minio) pods crash in a loop withPermission Deniederrors.
- The cluster fails to pull the newly built PipeCD image (
Root Cause
- Registry Hardcoding: The Makefile currently assumes a Kind registry setup and runs a hardcoded
docker push localhost:5001/...to publish images. Other clusters running on isolated runtimes (like containerd in K3s/K3d) cannot resolve this host registry out-of-the-box. - Storage Permissions: HostPath directories (e.g.
/tmp/pipecd-data/*) created by Kubernetes on the host system are owned byroot. However, the MySQL, Redis, and MinIO containers run as non-root users (UID 999 or 1000) and lack write permissions to these directories.
Expected Behavior
The local development setup should be platform-agnostic, supporting K3s, Minikube, K3d, and other distributions seamlessly alongside Kind.
Proposed Solution
- Dynamic Environment Detection in Makefile: Update
run/pipecdto inspect the active Kubernetes context.- For K3s/K3d, save the image and import it directly using
sudo k3s ctr --namespace k8s.io images import. - For Minikube, load the image using
minikube image load. - Fall back to
docker pushonly for Kind/standard setups.
- For K3s/K3d, save the image and import it directly using
- Permission Initialization: Add
initContainersto the Redis, MySQL, and MinIO template deployments indeployment.yamlthat run as root (runAsUser: 0) tochownthe mounted directories to the correct non-root UIDs before the main containers start.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Makefile target run/pipecd and the associated local Helm values to trace image publishing and host-path storage setup. Then inspect the Redis, MySQL, and MinIO deployment.yaml templates. Done means local setup works on Kind, K3s, K3d, Minikube, and other named distributions without image-pull or storage-permission failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, helm, kubernetes
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100