Azure-Samples / Azure-Samples/tutor

[P1] aca: fresh-environment container app writes time out with Operation expired

Open
#118 1 comment 0 reactions 0 assignees View on GitHub
backend component:aca component:ci-cd priority:P1 tracking type:bug type:performance
Dominant language
Python
Stars
17
Forks
8
PR merge metrics
No merged PRs in 30d

Description

## Problem
New-environment provisions repeatedly hit ACA revision provisioning timeouts across all backend apps (`Operation expired`) during `Microsoft.App/containerApps/write`.

## Type
- performance degradation
- bug

## Evidence
- Workflow run: https://github.com/Azure-Samples/tutor/actions/runs/23124471384
- Artifact: `aca-provision-diagnostics-178dev-23124471384`
- In `.tmp/aca-23124471384/activity/resource-group.activity-failures.json`, all 8 backend apps failed with same correlation id `133835aa44c069c72a688c956da3eb8d` and status message `ContainerAppOperationError ... Error details: Operation expired.`
- Per-app snapshot example `.tmp/aca-23124471384/apps/chat/app.json`:
- `provisioningState: Failed`
- `latestRevisionName: null`
- `latestReadyRevisionName: null`
- System log example `.tmp/aca-23124471384/logs/chat.system.log` shows repeated `ContainerAppUpdate` events and no app stack traces.

## Reproduction
1. Trigger `.github/workflows/azd-deploy.yml` with a fresh dev environment override (e.g., `178dev`).
2. Let Terraform create ACA environment and all backend apps in same provision operation.
3. Observe ACA write failures with `Operation expired` across multiple services.

## Acceptance Criteria
- [ ] Provision in a fresh env no longer exhibits multi-service ACA `Operation expired` failures in same correlation window.
- [ ] Add telemetry in workflow summary: count of ACA write failures by correlation id and service.
- [ ] Apply a deterministic mitigation (e.g., staged/serialized ACA app creation or bounded gating) validated in at least one fresh-env run.
- [ ] Document ACA-timeout triage checklist in runbook with required commands and artifact paths.

## Suggested Owner
`platform-quality` (implementation) + `system-architect` (orchestration constraints)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.