microsoft / microsoft/azure-container-apps
Managed identity image pull failure on replica restart 24 hours after initial start
Nobody has claimed this yet.
- Dominant language
- Bicep
- Stars
- 445
- Forks
- 54
- Avg merge
- 5h 14m
- Merged PRs (30d)
- 3
Description
This issue is a: (mark with an x)
- bug report -> please search issues before submitting
- documentation issue or request
- regression (a behavior that used to work and stopped in a new release)
Issue description
ACA with (user assigned) managed identity authorized with AcrPull to ACR. Initial container startup is fine, and also works if the replica crashes and restarts within a day. If the replica crashes and restarts 24 hours after initial start, the restart fails with an image pull error.
Microsoft.App/containerApps/revisions state:
- provisioningState is Failed
- runningState is Failed
- runningStateDetails is 1/3 Failed:xxxContainerCrashing | 1/3 Pending:ImagePullBackOff on legion | 1/3 Running
ContainerAppSystemLogs show repeating logs every 5 minutes:
- Pulling image 'xxx.azurecr.io/xxx:xxx'
- Container 'xxx' was terminated with exit code '' and reason 'ImagePullFailure'
In ACR ContainerRegistryRepositoryEvents there are no Pull operations logged as part of restart.
After restarting the revision things start working again, and this can be also seen as a successful image pull in ACR ContainerRegistryRepositoryEvents.
My assumption is that ACA is authenticating using MSI and acquiring a token for ACR authentication valid for 24 hours, but is not renewing the token after 24 hours has passed. Restarting the revision forces reacquisition of this token and image pull works.
Steps to reproduce
- ACA with managed identity authenticated image pull against ACR
- Wait 24 hours after initial start
- Container restart fails to pull image
Expected behavior
Managed identity based image pull should work on replica restart regardless of how much time has passed since initial start.
Actual behavior
Managed identity based image pull on replica restart fails 24 hours after initial start.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with an Azure Container App using a user-assigned managed identity and AcrPull access, then inspect the revision state and ContainerAppSystemLogs after a replica restarts 24 hours later. Compare ACR ContainerRegistryRepositoryEvents for the failed restart and a manually restarted revision; done means image pulls continue to work after the 24-hour boundary without manually restarting the revision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- authentication, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100