microsoft / microsoft/azure-container-apps

Managed identity image pull failure on replica restart 24 hours after initial start

Open
#960 39 comments 9 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Needs: triage :mag:
Dominant language
Bicep
Stars
445
Forks
54
Avg merge
5h 14m
Merged PRs (30d)
3

Description

This issue is a: (mark with an x)
  • bug report -> please search issues before submitting
  • documentation issue or request
  • regression (a behavior that used to work and stopped in a new release)
Issue description

ACA with (user assigned) managed identity authorized with AcrPull to ACR. Initial container startup is fine, and also works if the replica crashes and restarts within a day. If the replica crashes and restarts 24 hours after initial start, the restart fails with an image pull error.

Microsoft.App/containerApps/revisions state:

  • provisioningState is Failed
  • runningState is Failed
  • runningStateDetails is 1/3 Failed:xxxContainerCrashing | 1/3 Pending:ImagePullBackOff on legion | 1/3 Running

ContainerAppSystemLogs show repeating logs every 5 minutes:

  • Pulling image 'xxx.azurecr.io/xxx:xxx'
  • Container 'xxx' was terminated with exit code '' and reason 'ImagePullFailure'

In ACR ContainerRegistryRepositoryEvents there are no Pull operations logged as part of restart.

After restarting the revision things start working again, and this can be also seen as a successful image pull in ACR ContainerRegistryRepositoryEvents.

My assumption is that ACA is authenticating using MSI and acquiring a token for ACR authentication valid for 24 hours, but is not renewing the token after 24 hours has passed. Restarting the revision forces reacquisition of this token and image pull works.

Steps to reproduce
  1. ACA with managed identity authenticated image pull against ACR
  2. Wait 24 hours after initial start
  3. Container restart fails to pull image

Expected behavior
Managed identity based image pull should work on replica restart regardless of how much time has passed since initial start.

Actual behavior
Managed identity based image pull on replica restart fails 24 hours after initial start.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with an Azure Container App using a user-assigned managed identity and AcrPull access, then inspect the revision state and ContainerAppSystemLogs after a replica restarts 24 hours later. Compare ACR ContainerRegistryRepositoryEvents for the failed restart and a manually restarted revision; done means image pulls continue to work after the 24-hour boundary without manually restarting the revision.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
authentication, cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.