radius-project / radius-project/radius
Multicluster with containerImages: image pull credentials stored as Radius secrets land on the control-plane cluster, so target-cluster image pulls fail
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.7k
- Forks
- 137
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 110
Description
Bug information
In a multicluster deployment (control-plane cluster + external target cluster via RADIUS_TARGET_KUBECONFIG), a private container image requires a registry pull credential. If that credential is stored as a Radius secret resource, the underlying Kubernetes Secret is created on the control-plane cluster, while the container Deployment/Pod (and its imagePullSecrets reference) is created on the target cluster. The kubelet on the target cluster then cannot find the credential and the image pull fails.
Steps to reproduce
- Install Radius on a control-plane cluster and configure an external target cluster (set
RADIUS_TARGET_KUBECONFIGso application output resources deploy to the target cluster). - Store a private registry pull credential as a Radius secret resource (e.g.
Applications.Core/secretStores, or the equivalentRadius.Security/secrets). - Deploy an application with a container that references a private image and uses that secret as its image pull credential.
- Observe the container fail to start on the target cluster with an image pull / authentication error (
ImagePullBackOff).
Observed behavior
The registry credential exists as a Kubernetes Secret on the control-plane cluster, but the workload runs on the target cluster and its kubelet has no access to that secret, so authentication to the private registry fails.
Root cause — the write paths diverge between clusters:
-
Application output resources (container
Deployment, etc.) go to the target cluster. The async worker resolves the deployment processor against the target-cluster clients whenRADIUS_TARGET_KUBECONFIGis set:pkg/server/asyncworker.go:160—deployment.NewDeploymentProcessor(appModel, ..., clients.RuntimeClient, clients.ClientSet)whereclientscome fromdeploymentTargetClients().pkg/server/asyncworker.go:179-200—deploymentTargetClients()returns target-cluster clients whenRADIUS_TARGET_KUBECONFIGis set.- Multicluster v1 contract:
pkg/kubeutil/config.go(TargetKubeconfigEnvVar = "RADIUS_TARGET_KUBECONFIG").
-
Secret-store-managed Kubernetes Secrets go to the control-plane cluster. The frontend controller's
KubeClientis hardcoded to the control-plane client and is never swapped for the target cluster:pkg/server/asyncworker.go:152—KubeClient: k8s.RuntimeClient(control plane).pkg/corerp/frontend/controller/secretstores/kubernetes.go:228,244,293,295—UpsertSecret()creates/patches the namespace and creates/updates theSecretviaoptions.KubeClient(control plane).
Result: in multicluster mode the pull-credential Secret and the workload that consumes it are provisioned on different clusters.
Related gap: container rendering currently has no first-class way to wire a Radius secret to a Pod's imagePullSecrets — pkg/corerp/renderers/container/render.go sets container.Image (~line 348) and creates connection-data secrets via makeSecret() (~lines 896-913, scoped to options.Environment.Namespace), but does not populate imagePullSecrets from a secret store.
Desired behavior
When a container in a multicluster environment references a registry pull credential stored as a Radius secret, the credential must be materialized on the same (target) cluster where the workload runs, so the kubelet can authenticate and pull the image.
Potential fix directions to discuss:
- Introduce a per-resource target configuration (a "target config") so a resource — or at least container images / their pull secrets — can declare which cluster it belongs to, and ensure any secrets required by that resource are provisioned to the same target cluster.
- Alternatively, scope it narrowly to container images: ensure image pull secrets referenced by a container are always created on the target cluster alongside the
Deployment, rather than on the control-plane cluster.
Workaround
Manually create the registry pull secret directly on the target cluster (out of band) and reference it, rather than managing it through a Radius secret resource.
Additional context
Today the only target selection is global, via RADIUS_TARGET_KUBECONFIG (all application output resources go to one target cluster); there is no per-resource or per-environment target override for where secrets are written. KubernetesCompute only carries namespace/identity/resourceID, not a target cluster. This makes the divergence between control-plane secret writes and target-cluster workload writes structural rather than incidental.
Key files:
pkg/server/asyncworker.go(target routing + hardcoded control-planeKubeClient)pkg/corerp/frontend/controller/secretstores/kubernetes.go(UpsertSecret)pkg/corerp/renderers/container/render.go(container/image + secret rendering)pkg/kubeutil/config.go(RADIUS_TARGET_KUBECONFIGcontract)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read pkg/server/asyncworker.go and pkg/kubeutil/config.go first to trace target-client selection, then inspect UpsertSecret in pkg/corerp/frontend/controller/secretstores/kubernetes.go and rendering in pkg/corerp/renderers/container/render.go. Done means a Radius-managed image-pull credential is provisioned on the same target cluster as its workload, with multicluster behavior covered by validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100