Terraform: retry lifecycle recorder ADX principal assignment after managed identity propagation
- Dominant language
- Go
- Stars
- 15
- Forks
- 3
- Avg merge
- 21h 53m
- Merged PRs (30d)
- 131
Description
## Summary
Terraform deployment with `enable_lifecycle_recorder = true` can fail while granting the lifecycle recorder managed identity `Ingestor` access to the ADX Metrics database.
The failure occurs in `azurerm_kusto_database_principal_assignment.lifecycle_recorder`:
```
ResourceNotFound: Failed to run add principal assignment action for service Engine-. AAD principal was not found.
```
## Evidence
- Terraform created the user-assigned managed identity and its federated identity credential successfully.
- `az identity show` returned the expected client ID and principal ID.
- `az ad sp show --id ` succeeded.
- Terraform passes the managed identity client ID as `principal_id` with `principal_type = "App"`, matching the pattern used by the already successful Portal and adx-mon ADX assignments.
- A retry approximately 38 minutes after managed identity creation returned the same ADX Engine error.
This points to an ADX to Entra principal resolution propagation or readiness gap, not an invalid Terraform ID type.
## Impact
The base AKS, TauGrid, Portal, ADX, and adx-mon deployment succeeds. The second Terraform apply cannot complete the lifecycle recorder Helm upgrade, so durable lifecycle history remains unavailable and Portal correctly remains in `live-only` history mode.
## Proposed fix
1. Add a bounded readiness and retry strategy before or around the ADX database principal assignment.
2. Treat this specific transient `AAD principal was not found` response as retryable for a documented bounded interval.
3. Preserve the client ID based assignment contract. Do not change it to the managed identity object ID because existing successful assignments use client IDs.
4. Emit actionable diagnostics on final failure: identity client ID, identity creation timestamp, retry duration, and a safe re-run command.
5. Avoid an unbounded fixed sleep. A retry loop with exponential backoff and an explicit maximum duration is preferred.
## Acceptance criteria
- A fresh deployment with lifecycle recorder enabled completes without a manual second `terraform apply` under normal Entra propagation delay.
- A persistent ADX lookup failure fails within the documented bound and explains how to retry safely.
- The lifecycle recorder deployment becomes Ready and Portal reports durable lifecycle history when its dependencies are available.
- Existing deployments with lifecycle recorder disabled are unchanged.
Contributor guide
Research direction
Start by locating the Terraform resource azurerm_kusto_database_principal_assignment.lifecycle_recorder and the lifecycle recorder Helm upgrade path. Reproduce the failure with lifecycle recorder enabled, then review az identity show and az ad sp show results alongside the ADX error. Done means bounded retries handle normal propagation, persistent failures provide the documented diagnostics and rerun guidance, and disabled deployments remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, helm, kubernetes, terraform
- Domain
- cloud, databases, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100