microsoft / microsoft/aspire

Deployment pipeline should retry/detect readiness when a managed identity's AcrPull role assignment hasn't propagated yet

Open
#19,658 0 comments 0 reactions 1 assignee Claimed by @mitchdenny View on GitHub
area-deployment
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

## Description

Deployment E2E tests intermittently fail when a Container App tries to pull its image via a user-assigned managed identity immediately after the corresponding role assignment (ACR `AcrPull`) was created. Azure RBAC role assignments can take time to propagate, but the `aspire deploy` pipeline does not wait for or retry on this condition — it treats the pull failure as a terminal deployment error.

## Observed failure

Run: https://github.com/microsoft/aspire/actions/runs/32808769295/job/97685417529 (attempt 1, job `Deploy (Deployment.EndToEnd-NspStorageKeyVaultDeploymentTests)`)

```
ContainerAppOperationError: Failed to provision revision for container app 'server'. Error details: The following field(s) are either invalid or missing. Field
'template.containers.server.image' is invalid with details: 'Invalid value: "envacr22llec5lkxsbo.azurecr.io/server:aspire-deploy-20260825043648": unable to pull
image using Managed identity
/subscriptions/***/resourceGroups/e2e-nsp-react-32808769295-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/env_mi
-22llec5lkxsbo for registry envacr22llec5lkxsbo.azurecr.io';..
```

The managed identity and its `AcrPull` role assignment on the registry were created earlier in the same pipeline run, and the very next step that pulls the image fails because the RBAC grant hasn't propagated through Azure AD/ARM yet.

## Impact

This is a recurring, transient failure across the Deployment E2E suite whenever a Container App is deployed immediately after granting its managed identity `AcrPull` access. It produces false-negative CI signal and requires manual reruns.

## Suggested fix

Harden the `aspire deploy` pipeline/CLI logic so that steps depending on a freshly-created role assignment detect and retry on this specific transient condition instead of failing immediately:
- After creating a role assignment needed for image pull, retry the dependent container app provisioning step with backoff (or poll for role-assignment propagation) before treating the failure as terminal.
- Distinguish this transient "not yet propagated" class of error from genuine identity/permission misconfiguration so we don't mask real bugs, and cap retries with a reasonable timeout.

## Related

Filed as part of investigating the Deployment E2E Tests run for microsoft/aspire@80836ab (PR #19656). Not caused by that PR's changes — this is pre-existing, long-standing flakiness in the deployment pipeline's resource-readiness handling. See also #19657 (companion DNS propagation lag issue) — both stem from the same root gap: the deploy pipeline doesn't verify true readiness of dependent Azure resources before proceeding.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.