pipe-cd / pipe-cd/pipecd

bug(local-dev): local development setup fails on all non-Kind Kubernetes clusters (K3s, Minikube, K3d)

Open
#7,023 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug
Dominant language
Go
Stars
1.4k
Forks
365
Avg merge
1d 4h
Merged PRs (30d)
84

Description

What happened?

The local development environment (run via make run/pipecd and the associated local Helm values) is tightly coupled with Kind. When attempting to start the local development cluster on any non-Kind distribution (such as K3s, Minikube, K3d, or MicroK8s), the setup fails due to registry image pulling and persistent storage permission blockers.

Steps to Reproduce
  1. Configure your Kubernetes context to point to a local K3s or Minikube cluster.
  2. Enable local storage by setting localStorageVolumes.enabled: true in the Helm values.
  3. Run make run/pipecd.
  4. Observe:
    • The cluster fails to pull the newly built PipeCD image (ErrImagePull) because it cannot reach/resolve localhost:5001.
    • If the image is manually sideloaded, the database (mysql), cache (redis), and storage (minio) pods crash in a loop with Permission Denied errors.
Root Cause
  1. Registry Hardcoding: The Makefile currently assumes a Kind registry setup and runs a hardcoded docker push localhost:5001/... to publish images. Other clusters running on isolated runtimes (like containerd in K3s/K3d) cannot resolve this host registry out-of-the-box.
  2. Storage Permissions: HostPath directories (e.g. /tmp/pipecd-data/*) created by Kubernetes on the host system are owned by root. However, the MySQL, Redis, and MinIO containers run as non-root users (UID 999 or 1000) and lack write permissions to these directories.
Expected Behavior

The local development setup should be platform-agnostic, supporting K3s, Minikube, K3d, and other distributions seamlessly alongside Kind.

Proposed Solution
  1. Dynamic Environment Detection in Makefile: Update run/pipecd to inspect the active Kubernetes context.
    • For K3s/K3d, save the image and import it directly using sudo k3s ctr --namespace k8s.io images import.
    • For Minikube, load the image using minikube image load.
    • Fall back to docker push only for Kind/standard setups.
  2. Permission Initialization: Add initContainers to the Redis, MySQL, and MinIO template deployments in deployment.yaml that run as root (runAsUser: 0) to chown the mounted directories to the correct non-root UIDs before the main containers start.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Makefile target run/pipecd and the associated local Helm values to trace image publishing and host-path storage setup. Then inspect the Redis, MySQL, and MinIO deployment.yaml templates. Done means local setup works on Kind, K3s, K3d, Minikube, and other named distributions without image-pull or storage-permission failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, helm, kubernetes
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.