microsoft / microsoft/aspire

AKS environment does not support AcrPull for a cross-resource-group existing ACR

Open
#18,277 1 comment 0 reactions 0 assignees View on GitHub
area-deployment azure triage:bot-seen
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

## Summary

When an AKS compute environment (`AddAzureKubernetesEnvironment`) is pointed at an **existing Azure Container Registry that lives in a different resource group** (via `PublishAsExisting("myacr", "other-rg")` + `WithContainerRegistry(acr)`), the generated kubelet `AcrPull` role assignment does not actually target that cross-RG registry. The registry is referenced by **name only**, so the `existing` lookup resolves in the **AKS cluster's own resource group**, not the registry's real resource group.

This is the AKS analog of #11256. Unlike ACA/App Service (which produced a hard **BCP139** compile error), AKS **compiles successfully** but is functionally incorrect for the cross-RG case — making this a latent/silent limitation rather than a build break.

## Repro

```csharp
var acr = builder.AddAzureContainerRegistry("acr")
.PublishAsExisting("myexistingacr", "my-existing-resource-group");

builder.AddAzureKubernetesEnvironment("env")
.WithContainerRegistry(acr);

builder.AddProject("apiservice");
```

## What gets generated

The AKS environment bicep declares a **fresh local `existing` registry keyed off a name parameter** and scopes the kubelet `AcrPull` role to it:

```bicep
param acrName string // only the NAME is passed in

resource acr 'Microsoft.ContainerRegistry/registries@2025-04-01' existing = {
name: acrName // no `scope` -> resolves in the AKS deployment's RG
}

resource acrPullRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(acr.id, aks.id, subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '7f951dda-4ed3-4680-a7ca-43fe172d538d'))
properties: {
principalId: aks.properties.identityProfile.kubeletidentity.objectId
roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '7f951dda-4ed3-4680-a7ca-43fe172d538d')
principalType: 'ServicePrincipal'
}
scope: acr
}
```

(Confirmed by `tests/Aspire.Hosting.Azure.Kubernetes.Tests/Snapshots/AzureKubernetesEnvironmentExtensionsTests.AddLoadBalancer_BicepEnablesIngressProfileAndUsesPreviewApi.verified.bicep`.)

Because `acr` carries no `scope` property, ARM resolves `acrName` as a registry in the **AKS resource group**. For a registry that genuinely lives in `my-existing-resource-group`, that resource id does not exist in the AKS RG, so the role assignment is created against the wrong scope (or fails at deploy time).

## Why it differs from ACA/App Service (#11256)

- **ACA / App Service (old code):** referenced the actual `AzureContainerRegistryResource` symbol directly, which renders as an `existing` resource carrying a cross-RG `scope: resourceGroup('other-rg')`. A role assignment scoped to a cross-scope resource inside the same file triggers **BCP139** at compile time.
- **AKS:** never references the cross-RG registry symbol; it passes only `registry.Resource.NameOutputReference` as the `acrName` parameter and builds a same-scope local `existing` resource. No BCP139, but no cross-RG targeting either.

## Relevant code

`src/Aspire.Hosting.Azure.Kubernetes/AzureKubernetesEnvironmentExtensions.cs`:

- Line ~100 / ~356: `resource.Parameters["acrName"] = defaultRegistry.Resource.NameOutputReference;` (and the `WithContainerRegistry` override) — only the registry **name** is flowed, never its resource group.
- Lines ~674-709: the kubelet `AcrPull` role assignment block. It does `var acr = ContainerRegistryService.FromExisting("acr"); acr.Name = acrNameParam;` and `Scope = new IdentifierExpression(acr.BicepIdentifier)`. `acr.Scope` (the registry's resource group) is never set.

## Suggested fix direction

Apply the same approach used to fix #11256 for ACA/App Service: when the configured registry is an existing resource in a different resource group, emit the `AcrPull` role as a **separately-scoped role-assignment module** (`scope: resourceGroup('my-existing-resource-group')`) rather than an inline role against a name-only local `existing` resource. The kubelet identity principalId would need to flow into that module.

Alternatively, thread the registry's resource group through to the AKS bicep so the local `existing acr` can set its `scope` to the correct RG.

## Scope / impact

- Same-RG existing registries and auto-created default registries are unaffected (they live in the deployment RG).
- Only genuinely cross-RG existing registries are impacted.
- This is **out of scope** for the #11256 minimal fix (PR #18118), which intentionally left `Aspire.Hosting.Azure.Kubernetes` untouched because AKS does not hit the BCP139 compile error.

## Notes

The cross-RG deploy-time failure is inferred from standard ARM `existing`-resource scoping semantics (no `scope` => current resource group) and the generated bicep; it has not been empirically deploy-tested.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.