Incident Mitigation: 500 errors and resource scaling for Container App ca-grubify-api

Summary
- Incident: INC0010031 | Severity: 1
- Service: Azure Container Apps – ca-grubify-api (Add-to-cart not responding; HTTP 500s)
- Subscription: cbf44432-7f45-4906-a85d-d2b14a1e8328 | RG: rg-grubify-app
- Time window analyzed: last ~60 minutes (diagnostics blocked by RBAC)

Timeline (UTC)
- 08:44:26 – Incident acknowledged; baseline and revision retrieval attempted
- 08:45–08:47 – Container App info, revision, logs, and metrics retrieval returned empty/403 AuthorizationFailed (RBAC)
- 08:47:35 – RBAC issue documented; requested temporary Reader and Container App Contributor (or RG Contributor) to proceed with diagnostics/mitigation
- 08:50+ – Proceeded with repository/IaC review to prepare guidance while awaiting access

Observed/Constraints
- Could not access logs, AppHealthInfo, CPU/Memory, or request metrics due to AuthorizationFailed (403) for resource: /subscriptions/cbf44432-7f45-4906-a85d-d2b14a1e8328/resourceGroups/rg-grubify-app/providers/Microsoft.App/containerApps/ca-grubify-api
- Symptoms from incident: Persistent 500s; add-to-cart endpoint unresponsive. Typical causes include unhandled exceptions in controller/service, dependency timeouts (DB/Redis/inventory/pricing), or resource saturation leading to timeouts.

Recommended Code Fixes (based on add-to-cart patterns)
1) Harden the controller/service path
- Validate inputs (productId, quantity, user/session) and return 4xx for validation failures
- Guard against NullReference on cart/session and missing inventory/price data
- Ensure idempotency for add-to-cart to avoid duplicate operations under retries
2) Dependency resiliency
- Wrap DB/Redis/inventory calls with timeouts (1–3s) and bounded retries + exponential backoff (Polly or similar)
- Add circuit breaker to fail fast on sustained downstream issues and return friendly 503 with Retry-After
- Use cancellation tokens propagated from HTTP request
3) Error handling & observability
- Catch and map known exceptions to 4xx/5xx appropriately; avoid leaking 500 for expected conditions
- Log structured context (cartId, userId, productId, correlationId) and tag add-to-cart spans
- Emit metrics: add-to-cart count, success rate, p95 latency, dependency error counts

Operational/Mitigation Guidance (to be applied once RBAC unblocks)
- If sustained high memory or OOM suspected: scale to 4 CPU / 8Gi memory
az containerapp update --ids --cpu 4 --memory 8Gi --subscription
- If CPU >80% sustained: increase CPU stepwise (1->2, 2->4) and memory proportionally (2Gi->4Gi->8Gi)
- Ensure min replicas >= 2 for HA; set max replicas high enough for bursts (e.g., 10)
- Rollback example
az containerapp update --ids --cpu --memory Gi --min-replicas --max-replicas --subscription
- If scaling not needed: restart latest revision to clear transient faults

IaC/Config Suggestions
- Persist infra via azd (if used): azd infra synth; check infra/main.bicep or manifests/containerapp.tmpl.yaml
- Ensure Container App ingress, targetPort, and probe settings are correct
- Configure autoscaling rules on HTTP RPS/CPU/Memory; set requests/limits to prevent throttling
- Verify environment variables and connection strings for cart dependencies; set reasonable timeouts (2–5s) and retry policies via config

Follow-ups
- Once access is granted, we will:
1) Pull recent logs and revision logs to identify exact error signatures (HTTP 500 traces, timeouts, OOM, restarts)
2) Check CPU/Memory/requests and 5xx rates; correlate with deployment timeline
3) Apply mitigation (scale up or restart) and run a 60–100 min monitoring loop to confirm recovery

References
- Resource (pending RBAC): /subscriptions/.../resourceGroups/rg-grubify-app/providers/Microsoft.App/containerApps/ca-grubify-api
- Incident: INC0010031

Please triage to add-to-cart owners. Add resiliency and guardrails as above; we will append concrete log/metric evidence after RBAC is enabled.
---
*This issue was created by sreagent-octopets-007--70b460e3*
Tracked by the SRE agent [here](https://portal.azure.com/?feature.customPortal=false&feature.canmodifystamps=true&feature.fastmanifest=false&nocdn=force&websitesextension_loglevel=verbose&Microsoft_Azure_PaasServerless=betaµsoft_azure_paasserverless_assettypeoptions=%7B%22SreAgentCustomMenu%22%3A%7B%22options%22%3A%22%22%7D%7D#view/Microsoft_Azure_PaasServerless/AgentFrameBlade.ReactView/id/%2Fsubscriptions%2Fca5ce512-88e1-44b1-97c6-22caf84fb2b0%2FresourceGroups%2Frg-octopets-v2%2Fproviders%2FMicrosoft.App%2Fagents%2Fsreagent-octopets-007/sreLink/%2Fviews%2Factivities%2Fthreads%2F7ca3eb58-e76d-42cf-a604-b26fb559eecc)

Start with infra/main.bicep or manifests/containerapp.tmpl.yaml and the add-to-cart controller/service entry point, then obtain Azure Container App logs and metrics once RBAC is available. Confirm the root cause before changing scaling or resiliency settings; done means the 500s are resolved and post-change monitoring confirms recovery.

Written by the indexing model from the issue text.