radius-project / radius-project/radius

Multicluster with containerImages: image pull credentials stored as Radius secrets land on the control-plane cluster, so target-cluster image pulls fail

Open
#12,331 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug triaged
Dominant language
Go
Stars
1.7k
Forks
137
Avg merge
2d 17h
Merged PRs (30d)
110

Description

Bug information

In a multicluster deployment (control-plane cluster + external target cluster via RADIUS_TARGET_KUBECONFIG), a private container image requires a registry pull credential. If that credential is stored as a Radius secret resource, the underlying Kubernetes Secret is created on the control-plane cluster, while the container Deployment/Pod (and its imagePullSecrets reference) is created on the target cluster. The kubelet on the target cluster then cannot find the credential and the image pull fails.

Steps to reproduce
  1. Install Radius on a control-plane cluster and configure an external target cluster (set RADIUS_TARGET_KUBECONFIG so application output resources deploy to the target cluster).
  2. Store a private registry pull credential as a Radius secret resource (e.g. Applications.Core/secretStores, or the equivalent Radius.Security/secrets).
  3. Deploy an application with a container that references a private image and uses that secret as its image pull credential.
  4. Observe the container fail to start on the target cluster with an image pull / authentication error (ImagePullBackOff).
Observed behavior

The registry credential exists as a Kubernetes Secret on the control-plane cluster, but the workload runs on the target cluster and its kubelet has no access to that secret, so authentication to the private registry fails.

Root cause — the write paths diverge between clusters:

  • Application output resources (container Deployment, etc.) go to the target cluster. The async worker resolves the deployment processor against the target-cluster clients when RADIUS_TARGET_KUBECONFIG is set:

    • pkg/server/asyncworker.go:160deployment.NewDeploymentProcessor(appModel, ..., clients.RuntimeClient, clients.ClientSet) where clients come from deploymentTargetClients().
    • pkg/server/asyncworker.go:179-200deploymentTargetClients() returns target-cluster clients when RADIUS_TARGET_KUBECONFIG is set.
    • Multicluster v1 contract: pkg/kubeutil/config.go (TargetKubeconfigEnvVar = "RADIUS_TARGET_KUBECONFIG").
  • Secret-store-managed Kubernetes Secrets go to the control-plane cluster. The frontend controller's KubeClient is hardcoded to the control-plane client and is never swapped for the target cluster:

    • pkg/server/asyncworker.go:152KubeClient: k8s.RuntimeClient (control plane).
    • pkg/corerp/frontend/controller/secretstores/kubernetes.go:228,244,293,295UpsertSecret() creates/patches the namespace and creates/updates the Secret via options.KubeClient (control plane).

Result: in multicluster mode the pull-credential Secret and the workload that consumes it are provisioned on different clusters.

Related gap: container rendering currently has no first-class way to wire a Radius secret to a Pod's imagePullSecretspkg/corerp/renderers/container/render.go sets container.Image (~line 348) and creates connection-data secrets via makeSecret() (~lines 896-913, scoped to options.Environment.Namespace), but does not populate imagePullSecrets from a secret store.

Desired behavior

When a container in a multicluster environment references a registry pull credential stored as a Radius secret, the credential must be materialized on the same (target) cluster where the workload runs, so the kubelet can authenticate and pull the image.

Potential fix directions to discuss:

  • Introduce a per-resource target configuration (a "target config") so a resource — or at least container images / their pull secrets — can declare which cluster it belongs to, and ensure any secrets required by that resource are provisioned to the same target cluster.
  • Alternatively, scope it narrowly to container images: ensure image pull secrets referenced by a container are always created on the target cluster alongside the Deployment, rather than on the control-plane cluster.
Workaround

Manually create the registry pull secret directly on the target cluster (out of band) and reference it, rather than managing it through a Radius secret resource.

Additional context

Today the only target selection is global, via RADIUS_TARGET_KUBECONFIG (all application output resources go to one target cluster); there is no per-resource or per-environment target override for where secrets are written. KubernetesCompute only carries namespace/identity/resourceID, not a target cluster. This makes the divergence between control-plane secret writes and target-cluster workload writes structural rather than incidental.

Key files:

  • pkg/server/asyncworker.go (target routing + hardcoded control-plane KubeClient)
  • pkg/corerp/frontend/controller/secretstores/kubernetes.go (UpsertSecret)
  • pkg/corerp/renderers/container/render.go (container/image + secret rendering)
  • pkg/kubeutil/config.go (RADIUS_TARGET_KUBECONFIG contract)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read pkg/server/asyncworker.go and pkg/kubeutil/config.go first to trace target-client selection, then inspect UpsertSecret in pkg/corerp/frontend/controller/secretstores/kubernetes.go and rendering in pkg/corerp/renderers/container/render.go. Done means a Radius-managed image-pull credential is provisioned on the same target cluster as its workload, with multicluster behavior covered by validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.