Azure / Azure/azure-functions-host

Host reports 403 AuthorizationPermissionMismatch on Blobs with correct RBAC, indexes no functions, and reports nothing to App Insights

Open
#11,938 5 comments 0 reactions 0 assignees View on GitHub
Needs: Triage (Functions)
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

### Investigative information

- **Timestamp**: 2026-08-21, ~09:20–13:55 UTC
- **Function App version**: host `4.1052.300.0`, `FUNCTIONS_EXTENSION_VERSION=~4`
- **Region**: West Europe
- **Runtime**: `NODE|24`, Linux, `kind: functionapp,linux`
- **Plan**: Basic B1 (dedicated), `alwaysOn: true`
- **Storage connection**: identity-based, **system-assigned** managed identity

### Summary

A Function App host reaches `state: Error` with a single 403 `AuthorizationPermissionMismatch`
against Blob Storage, and therefore **indexes no functions at all** — `az functionapp function list`
returns empty — while the site reports `Running` and the package is deployed on disk.

The identity holds `Storage Blob Data Owner`, `Storage Queue Data Contributor` and `Reader` on the
target storage account, with **no ABAC conditions** and **no deny assignments**. The host's own
internal containers (`azure-webjobs-hosts`, `azure-webjobs-secrets`) **do exist** in that account,
which means blob access worked at least once.

**The same ARM template, same runtime, same role set, deployed to a different subscription, works
correctly and indexes its function.**

### Repro steps

1. Deploy, from one ARM/Bicep template, a Linux Function App on a **dedicated B1** plan alongside a
storage account, with `AzureWebJobsStorage` configured for identity:
- `AzureWebJobsStorage__accountName`
- `AzureWebJobsStorage__blobServiceUri`
- `AzureWebJobsStorage__queueServiceUri`
2. In the same deployment, create role assignments for the app's **system-assigned** identity on the
storage account: `Storage Blob Data Owner`, `Storage Queue Data Contributor`, `Reader`.
3. Deploy code with `az functionapp deployment source config-zip` (a pre-built bundle; no remote build).
4. Restart the app and query `GET /admin/host/status`.

### Expected behavior

The host starts and indexes the functions declared by the deployed package (a single blob-triggered
function, Node v4 programming model).

### Actual behavior

```
state: Error
version: 4.1052.300.0
errors: 1
Azure.Storage.Blobs: This request is not authorized to perform this operation using this permission.
RequestId: 535edfec-801e-00d8-7269-310f07000000
Time: 2026-08-21T12:32:06.7484680Z
Status: 403 (This request is not authorized to perform this operation using this permission.)
ErrorCode: AuthorizationPermissionMismatch
```

Further RequestIds with the same error, in case they help correlate server side:
`6d57152f-901e-00eb-2f61-3150ac000000` (11:36:11Z), `bc5daf1a-801e-0015-3879-2bcc07000000`.

Container log shows a repeating cycle:

```
Creating container with image: azure-functions/noble/linuxdedicated:4.1052.300-0-node24-appservice
Container is running.
Site startup probe succeeded after 65.5 seconds.
Site is running with deployment version:
Site started.
Container is terminating. Grace period: 5 seconds. <-- ~21 s after "Site started"
Container is terminated.
Site: stopped.
```

### What the error does **not** tell us

The message names `Azure.Storage.Blobs` but not the operation or the URI, so there is no way from the
outside to know which permission is missing. That is the core of this report: with `Storage Blob Data
Owner` + `Reader` granted, no conditions and no deny assignments, **which blob operation is being
rejected?**

### Ruled out, with evidence

| Hypothesis | How it was ruled out |
|---|---|
| Missing data-plane roles | all three present, listed by `--scope` with their assignment IDs |
| ABAC conditions on the assignments | `condition` is `null` on all four assignments on the account |
| Deny assignments | `GET .../providers/Microsoft.Authorization/denyAssignments` on the account returns `[]` |
| Creation path (per #10970) | `azure-webjobs-hosts` and `azure-webjobs-secrets` **exist** — so container creation succeeded |
| Storage account configuration | **identical across 11 fields** to the working environment: `networkRuleSet.defaultAction: Allow`, `bypass: None`, no IP/VNet rules, `allowSharedKeyAccess`, `publicNetworkAccess`, `defaultToOAuthAuthentication`, `minimumTlsVersion`, `allowBlobPublicAccess`, `allowCrossTenantReplication`, `kind`, `sku` |
| VNet integration | none configured; no VNets in the resource group |
| Package not deployed | SCM deployment history: status 4 (Success) ×3; rsync of 1,244 files / 12.45 MB reported "Completed successfully" |
| Missing app settings | full app-setting name set identical to the working environment |
| Unsupported runtime | the working environment runs the same `NODE|24` and indexes fine |
| Contention on the shared plan | the co-located web app was deployed and stabilised; symptom unchanged |
| Deployment order | reproduced with the code deployed as the very last operation |
| Application Insights as a diagnostic | **unusable here**: the same query returns zero rows in the *working* environment too, so its emptiness proves nothing. `/admin/host/status` was the only source that reported anything |

### Two observations that may matter

**1. `Get Blob Service Properties` is not covered by the data roles.** We add `Reader` deliberately,
because that service-level read is required at startup and `Storage Blob Data Owner` does not grant
`Microsoft.Storage/storageAccounts/blobServices/read`. The documented guidance suggests `Storage
Account Contributor`, which we avoid because it includes `listkeys/action` — the exact credential path
that identity-based connections exist to remove. If `Reader` is genuinely insufficient for some other
startup operation, it would help enormously to have that documented, or to have a narrower built-in
role for it.

**2. The failure is completely silent from a monitoring standpoint.** No telemetry, no failed
executions, `state: Running` on the site, and a green deployment. The only way we found to detect it
was polling `/admin/host/status` from a scheduled job. This is the same complaint as #10970, which was
closed — the observability gap it describes is still very much present in `4.1052.300.0`.

### Related

- #10970 — same class of silent failure. Our containers exist, so this is not the creation path, but the
observability problem described there is exactly what made this take a full day to characterise.
- #8189 — ARM template redeploys wiping deployment-tool state and making functions disappear. Different
mechanism (consumption + `WEBSITE_RUN_FROM_PACKAGE`), same family of problem.

Contributor guide

Open the contributing guide

Research direction

Start with GET /admin/host/status and compare the failing and working subscriptions using the reported Azure.Storage.Blobs 403 RequestIds, identity roles, and startup configuration. Trace the host startup path that performs the rejected blob operation and verify the dedicated B1 deployment with the Node v4 package. Done means the host indexes the deployed function, identifies or documents the required permission, and exposes a useful failure signal.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, node.js
Domain
authentication, backend, cloud, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.