Managed identity cannot create Cosmos containers: data-plane RBAC has no create action, azd deploy never runs Bicep, and deployer container lists have drifted

## Issue

Deployments configured with `AZURE_COSMOS_AUTHENTICATION_TYPE=managed_identity` cannot create Cosmos DB containers. Whenever a container the application expects does not already exist, startup fails with a 403 at module import time — before Flask binds a port — and the App Service container never starts.

Previously reported in #693 (seen on both initial deployment and updates; worked around by switching to key auth). Related: #484. Not the same as #536, which fails earlier during `CosmosClient` construction and looks like endpoint resolution rather than RBAC.

## Root cause

Three compounding causes.

### 1. Container creation cannot be expressed in Cosmos data-plane RBAC

`application/single_app/config.py` builds the schema at module import time against the **data plane**:

```python
cosmos_database = cosmos_client.create_database_if_not_exists(cosmos_database_name)
# ... followed by 67 create_container_if_not_exists calls
```

These requests go to `https://.documents.azure.com`, which evaluates only the account's native `sqlRoleAssignments`. Per the [data plane security reference](https://learn.microsoft.com/en-us/azure/cosmos-db/reference-data-plane-security), the complete dataAction set is:

- `Microsoft.DocumentDB/databaseAccounts/readMetadata`
- `.../sqlDatabases/containers/executeQuery`, `readChangeFeed`, `executeStoredProcedure`, `manageConflicts`
- `.../sqlDatabases/containers/items/*`

There is no create action for databases or containers. Even the `containers/*` wildcard is documented as covering only *"executing queries, reading the change feed, managing conflicts, and executing stored procedures"*. And native RBAC does not support `notDataActions` — *"any action that isn't specified as an allowed dataAction is excluded automatically"* — so a **custom** `az cosmosdb sql role definition` cannot express it either.

Creation is an Azure RBAC **Action**, evaluated by ARM ([permissions reference](https://learn.microsoft.com/en-us/azure/role-based-access-control/permissions/databases)):

```
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/write Create a SQL database.
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/write Create or update a SQL container.
```

Consequences:

- **No Cosmos data-plane role, built-in or custom, can authorize the current code path.**
- Granting the managed identity `Cosmos DB Operator` or `Contributor` also does nothing, because the SDK request still goes to the data-plane endpoint. This is an easy dead end to spend a day in.
- The guidance recorded on #693 — run `cosmosDb-postDeployPerms.sh` — cannot fix it for two independent reasons: the script grants the **signed-in user**, not the App Service identity, and it grants data-plane Data Contributor, which lacks the action regardless.

The failure is intermittent-looking because `create_container_if_not_exists` reads first and only creates on 404. With `readMetadata`, containers that already exist read back fine — so managed identity appears to work whenever the schema happens to be complete.

### 2. `azd deploy` never provisions containers

Containers are created in exactly two places:

| Path | Mechanism | Plane | Runs on |
|---|---|---|---|
| `azd provision` / `azd up` | `cosmosDb.bicep` `cosmosContainers` loop | Control (ARM, deployer identity) | provision only |
| App startup | `create_container_if_not_exists` | Data (`documents.azure.com`) | key auth only |

`azd deploy` runs neither. Its only hook is `predeploy`, which builds and pushes the container image. `postconfig.py` performs `get_container_client` / `read_item` / `upsert_item` against `settings` only, and creates no schema.

So on the upgrade path, the application's own creation call is the **only** mechanism that can add a new container — and that call is authorized solely by the account key.

**Every release that adds a container is therefore a breaking change for every managed-identity deployment.** Key-based deployments self-heal silently on first boot, which is why this has remained invisible; managed identity is opt-in, since `AZURE_COSMOS_AUTHENTICATION_TYPE` defaults to `key`.

### 3. Deployer container lists have drifted from `config.py`

Even a full `azd provision` does not currently produce a complete schema:

| Source | Containers | Missing |
|---|---|---|
| `application/single_app/config.py` (runtime truth) | 67 | — |
| `deployers/bicep/modules/cosmosDb.bicep` | 63 | **4** |
| `deployers/terraform/main.tf` | 60 | **7** |
| `deployers/azurecli/deploy-simplechat.ps1` | 58 | **10** |

Missing from Bicep:

| Container | Partition key |
|---|---|
| `orchestration_runs` | `/conversation_id` |
| `orchestration_run_steps` | `/run_id` |
| `document_access_index` | `/scope_key` |
| `key_vault_secret_reminders` | `/scope_key` |

Terraform additionally omits `custom_pages`, `governance_policies`, `governance_item_policies`.

Partition keys on the containers that *are* present all match, so this is purely additive drift — containers were added to `config.py` without being added to the deployers.

`orchestration_runs` is the **13th** `create_container_if_not_exists` call in `config.py`, so a fresh Bicep + managed identity deployment dies there on first boot with a correct-looking role assignment in place.

## Impact

- Managed identity is effectively unusable for any deployment that upgrades, and unusable on fresh Bicep deployments today because of the drift.
- Failure mode is a crash loop with no port binding, which reads as a platform or networking fault rather than a permissions one.
- The visible evidence actively misleads: the `sqlRoleAssignment` is present and correct, so operators conclude permissions are fine.
- Only workarounds are switching to key auth or hand-creating containers, which blocks key-free/`disableLocalAuth` customer environments entirely.

## Proposed fix

### 1. Provision through ARM at runtime when using managed identity

When `AZURE_COSMOS_AUTHENTICATION_TYPE=managed_identity`, create missing databases and containers via `azure-mgmt-cosmosdb` (`CosmosDBManagementClient.sql_resources.begin_create_update_sql_container`) instead of the data-plane `create_container_if_not_exists`. Continue using the data-plane client for all item reads and writes.

This is the only option that works on the `azd deploy` path, because it lives in application code rather than infrastructure.

Requires granting the App Service identity control-plane actions. Prefer a least-privilege custom role over the broad built-in `Cosmos DB Operator`:

```
Microsoft.DocumentDB/databaseAccounts/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/write
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/operationResults/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/write
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/operationResults/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/throughputSettings/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/throughputSettings/read
```

Deliberately excludes delete, `listKeys`, `listConnectionStrings`, and `sqlRole*` management so the identity cannot escalate itself. This must be added to all three deployers so new deployments receive it automatically.

**Gating and schema versioning.** A naive implementation issues 68 ARM calls per Gunicorn worker on every boot. ARM is rate-limited and materially slower than the data plane, so this needs to be bounded:

- Derive a deterministic hash of the expected container set (names, partition keys, TTLs, indexing policies) at import time.
- Persist the applied hash as a schema-version marker document in the `settings` container.
- Skip reconciliation entirely when the stored hash matches — the steady-state boot cost becomes one point read.
- Reconcile only when the hash differs, and take a short-lived lease or use optimistic concurrency on the marker so concurrent workers do not stampede ARM during a rolling restart.
- Reconciliation must be additive only. It must never delete or replace a container, so an unexpected hash mismatch can never destroy data.
- Failure to reconcile should surface a clear, actionable startup error naming the missing container and the required action, rather than a raw 403.

Open questions worth resolving during design:

- Should reconciliation be gated behind an explicit opt-in setting for operators who prefer infrastructure to own schema entirely?
- Should the key-auth path also move to ARM for consistency, or keep the existing data-plane behaviour to avoid requiring control-plane RBAC on key-based deployments?
- Where should the marker live before the `settings` container itself exists — bootstrap ordering needs care.

### 2. Close the deployer gap

Add the missing containers to all three deployer lists: Bicep +4, Terraform +7, azurecli +10. Still required for fresh provisions and for key-based deployments, independently of item 1.

### 3. Drift test

Add a functional test asserting all three deployer lists match `config.py` on names, partition keys, and TTLs. Without this, item 2 rots again on the next feature that adds a container — the drift is the root cause here, not the symptom.

### 4. Correct the misleading guidance

`cosmosDb-postDeployPerms.sh` and the equivalent block in `azure.yaml` grant the **signed-in user**, not the application identity. The comment on #693 recommends this script as the fix; that should be corrected, and the managed identity documentation updated to state plainly that container creation is a control-plane operation.

## Acceptance criteria

- [ ] A managed identity deployment with an incomplete schema starts successfully and creates the missing containers.
- [ ] `azd deploy` of a build that adds a new container succeeds on a managed identity deployment without running `azd provision` and without key auth.
- [ ] Steady-state boot with an unchanged schema issues no container-provisioning ARM calls (verified by the schema-version marker short-circuit).
- [ ] Concurrent worker startup does not produce duplicate or conflicting provisioning calls.
- [ ] Reconciliation never issues a delete or replace against an existing container.
- [ ] Bicep, Terraform, and azurecli each provision all 67 containers with partition keys and TTLs matching `config.py`.
- [ ] A functional test fails when any deployer list drifts from `config.py`.
- [ ] The control-plane role is granted by all three deployers for managed identity deployments.
- [ ] A managed identity failure to provision produces an actionable startup error naming the container and required permission.
- [ ] Managed identity documentation states that container creation is control-plane, and the `postDeployPerms` guidance is corrected.

## Notes

- `deployers/Initialize-CosmosManagedIdentityAccess.ps1` (added in deployer `1.0.27`) is the remediation path for deployments already broken by this. It grants both planes, validates that the supplied principal is an object ID rather than a client ID, and creates all 67 containers via ARM. It is a stopgap, not the fix.
- A frequent secondary cause of "the role is assigned but nothing works": Cosmos stores any GUID in a `sqlRoleAssignment` without validating it, so an application (client) ID pasted where the object ID belongs produces an assignment that looks correct in the portal and CLI but grants nothing. Worth calling out in the docs.
- Counts above were verified by parsing `config.py`, `cosmosDb.bicep`, `main.tf`, and `deploy-simplechat.ps1` directly rather than by inspection.