Production environment serves two deployments simultaneously from the default hostname; ratio oscillates and never converges

## Summary

A production environment serves **two different deployments simultaneously** from the default `*.azurestaticapps.net` hostname. Which one a request receives appears to be per-request. The share served the current build **oscillates rather than converging** — 27% → 78% → 40% within minutes, with no deployment occurring in between — and requests are answered from artifact sets over two hours and two deployments old.

Every deployment reports `Deployment Complete`. ARM reports the environment `Ready` with a current `lastUpdatedOn` throughout.

This is on the **default hostname**, with no custom domain, no CDN, and no enterprise-grade edge in the path.

Standard SKU, East Asia. The hostname is redacted below; I'm happy to provide it privately to anyone at Microsoft who wants to reproduce this directly.

## Reproduction

Each deployment stamps `/build-id.txt` with its GitHub Actions run id, and a route rule makes that one path anonymous with `Cache-Control: no-store`. Every request below uses a **unique query string**, so no cache anywhere can return a stored copy.

```bash
U=https://.azurestaticapps.net/build-id.txt
for i in $(seq 1 40); do
rm -f /tmp/b
curl -s -o /tmp/b --max-time 10 -w '%{http_code} ' "$U?cb=$i-$RANDOM"
tr -d '\r\n' < /tmp/b; echo
done | sort | uniq -c | sort -rn
```

Expected: 40 identical lines. Actual, six minutes after a successful deployment:

```
21 200 30511573524-1 <- current build
15 200 30510492511-1 <- two deployments and 30 minutes old
3 000 <- no response
1 503
```

## Two builds, named, in a single 20-request sample

Taken by the deployment workflow itself immediately after reporting success:

```
11 200 30503495726-1 <- the deployment from 18 minutes earlier
9 200 30504402496-1 <- the deployment that had just completed
```

One URL, one moment, two different bodies, both `200`.

## It oscillates rather than converging

Sampling the same URL after a deployment at 03:11 UTC on 2026-07-30. **No deployment occurred between these samples.**

| UTC | current build | previous build | transport errors |
|---|---|---|---|
| 03:13:44 | 17/20 | 0 | 3× `500` |
| 03:20:36 | 11/40 (27%) | 29/40 | — |
| 03:22:10 | 31/40 (78%) | 9/40 | — |
| 03:52:51 | 16/40 (40%) | 24/40 | — |

The share served the new build fell to zero and recovered, twice, with nothing deployed.

## `staticwebapp.config.json` is affected independently of content

After a different deployment, the stale responses were not old content but **old routing rules**. A route granting anonymous access to `/build-id.txt` was present in the deployed config; a share of replicas did not have it and returned `302 -> /login` via the `401` `responseOverride` instead:

| Minutes after deployment | `200` | `302` (previous config) |
|---|---|---|
| 7 | 29/40 | 11/40 (27%) |
| 13 | 24/40 | 16/40 (40%) |
| 25 | 18/20 | 2/20 (10%) |

So the artifact set and the configuration can land on different replicas independently.

## Platform-generated errors while ARM reports `Ready`

- `500` on 3 of 20 requests, served from the Azure Static Web Apps branded error page
- `502` — "Web server received an invalid response while acting as a gateway or proxy server"
- Connections dropped with no response at all, up to 4 in 40

## User-visible impact

Mixing is per-request, not per-session, so a single page load fetches its HTML from one replica and its hashed JS chunks from another. When those come from different builds the chunk filenames don't match and the page 404s.

For an app with `rolesSource` and route-level `allowedRoles`, this presents as **fresh sign-ins failing intermittently**: authentication completes (Entra logs `errorCode 0`), `/.auth/complete` completes, and then the post-login landing page 404s. Existing session cookies keep working, so it goes unnoticed for hours and looks random. Two users hit this before we understood what we were looking at.

## Ruled out by direct measurement

| | |
|---|---|
| Application code | Rolled production back to the last known-healthy tree — stayed broken. The suspect commit signs in correctly on three separate staging environments. |
| Any cache — browser, CDN, or platform edge | Unique query string per request; 15 of 40 still returned a superseded build. Response carries `cache-control: no-store`. |
| Custom domain / third-party CDN | Not in the path. All measurements are against the default hostname. |
| Deployment not landing, or wrong `app_location` / `output_location` | The new build **is** served, by 21 of those 40 requests. A path fault would fail uniformly rather than split. |
| Managed functions | `/build-id.txt` is a static file; no function participates in serving it. The stale-config case is a route rule, also not function-served. |
| Deploy cadence | Two weeks in April sustained ~2 production deploys/day with no incidents — a higher rate than the incident window. |
| App settings / app registration | Byte-identical to the staging environments that work. No resource changes in the activity log for 7 days. |
| Leftover preview environments | Deleted all of them (`az staticwebapp environment list` → `default` only) and re-measured: still split 25/14. |
| The resource itself | The same fault occurred on a **different** Static Web App, in a different region and resource group, days earlier. We migrated specifically to escape it. |

## Mitigations attempted, none effective

- **Redeploy** — the product group's mitigation on #1281. Two consecutive deployments both failed verification; still split 21 minutes later.
- **Deploy the identical artifact twice per run** — on the theory that if current and previous are byte-identical, a lagging replica serves the same bytes. Replicas turn out to be stuck on artifact sets **far further back than one deployment**, so this bought nothing. Reverted.
- **`Cache-Control: no-store`** — already set and returned. The reporter on #1281 had `globalHeaders` `no-store` and still received a previous deployment's `index.html`.
- **Deleting preview environments** — suggested on a related Q&A thread. No effect.
- Not attempted, on the strength of #1766: delete and recreate. That reporter saw the fault reproduce on brand-new resources and concluded the stale state is keyed to the domain rather than the resource. [MS Q&A 5910175](https://learn.microsoft.com/en-us/answers/questions/5910175/) additionally documents resource-name reuse causing traffic to split between a populated and an empty environment.

## Possible correlation with the deployment client

`StaticSitesClient` self-updates and its version is printed in every run log. Ours:

| Deployment | Client | Outcome |
|---|---|---|
| 2026-07-21 | `20260714.240.1` | last healthy deployment |
| 2026-07-28 | `20260720.247.2` | **first broken deployment — never recovered** |
| 2026-07-29 → 30 | `20260720.247.2` | broken |
| 2026-07-30 | `20260726.254.1` | broken |

Nothing was deployed between 21 and 28 July and the site served normally throughout. Rolling the code back on 28 July did not help — the deployment client is the one variable a rollback cannot undo.

#1778 reports a different symptom (Next.js function provisioning failing outright) but names **the same version pair and the same last-good date of 2026-07-21**: *"The only delta between the last success and the failures is the deployment client version printed in the action logs."*

This is correlation across two organisations, not a claim of causation. Flagging it because it is checkable on your side and we cannot pin the client version to test it ourselves.

## Prior art

**#1281 (September 2023)** is the same fault, and Microsoft confirmed it privately at the time:

> "if I spam the reload button of a browser on our site 10 times within 10 seconds it could return 5 404 errors and 5 200 successes … it's also clear that **the SWA could randomly return a different index.html file which most likely belonged to a previous deployment of the SWA**"

> "**I got confirmation from Azure that the fault is indeed on their end.**"

Eight organisations in that thread. Resolved server-side within about five days. No status-page notification was issued.

**#1563** (open since 2024-10-21) describes the same serving behaviour — *"as if it's being load balanced, and you don't always get the same result"* — and has had no maintainer response in 21 months.

**Documented behaviour** ([MS Q&A 625770](https://learn.microsoft.com/en-us/answers/questions/625770/can-you-shed-some-light-on-enviroments-swap-in-azu), Microsoft moderator, 2021-11-22):

> "Static Web App deployments are atomic and we won't shift to serving the new content until it is readily available for all edge nodes that can serve it. **There should never be a time where a user is getting files from different commits when requesting the files directly.**"

That statement does not appear in the product documentation, and the observed behaviour contradicts it.

## Questions

1. Why does one environment serve two artifact sets, and two configurations, after `Deployment Complete`?
2. What bounds replica lag? We observe artifact sets more than two hours and two deployments old.
3. Why does the share of requests served the new build **fall** over time rather than rise to 100%?
4. What accounts for the `500`, `502` and dropped connections while ARM reports `Ready`?
5. Did anything in the content-distribution path change between `StaticSitesClient` `20260714.240.1` and `20260720.247.2`? **Is there a supported way to pin the deployment client version?**
6. Was there an unpublished Static Web Apps incident between 2026-07-20 and 2026-07-30? Eleven issues were filed in this repository in that window from multiple organisations, and the Azure status history shows nothing for the service.
7. What forces convergence, given that redeployment does not, and that #1766 reports recreation reproducing the fault?

## Environment

- Azure Static Web Apps, **Standard** SKU, **East Asia**
- Default `*.azurestaticapps.net` hostname; no custom domain, no CDN, no enterprise-grade edge, no private endpoint in the measurement path
- `Azure/static-web-apps-deploy@v1`, GitHub Actions, deployment token auth, `skip_app_build: true`
- VitePress static output plus one managed function used as `rolesSource`
- `staticwebapp.config.json` with `allowedRoles` route rules and `responseOverrides`
- Reproducible on demand as of 2026-07-31

Start with the curl loop against /build-id.txt and compare its results with the deployment workflow logs, including the StaticSitesClient versions. Read the evidence around #1281, #1563, #1766, and #1778, plus the linked Microsoft Q&A reports. Done means identifying the serving or deployment cause and documenting a verified fix or supported mitigation that makes responses converge on one deployment.

Written by the indexing model from the issue text.