cloudflare / cloudflare/vinext

`--experimental-warm-cdn-cache` re-caches the previous version on workers.dev (version-override header ignored), extending staleness instead of fixing it

Open
#2,592 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
8.8k
Forks
406
Avg merge
2d 6h
Merged PRs (30d)
120

Description

# `--experimental-warm-cdn-cache` re-caches the *previous* version on workers.dev (version-override header ignored), extending staleness instead of fixing it

## Summary

On a plain workers.dev deployment, `vinext-cloudflare deploy --experimental-warm-cdn-cache` reports `CDN warmup: 1/1 path(s) warmed.` and exits successfully, but the public URL keeps serving the **previous** build until the edge-cache TTL expires. Worse, the warmup request itself can create a **fresh cache entry containing the old version's HTML** seconds before the new version is promoted — resetting the TTL clock and making the stale window *longer* than if the flag had not been used.

Root cause: the pre-promotion warmup pins the staged (0% traffic) version with a `Cloudflare-Workers-Version-Overrides` header, but that header appears to be ignored on workers.dev, so the warmup request is served by the old version that still holds 100% of traffic — and that response is what gets cached.

## Environment

- `vinext` 1.0.0-beta.1, `@vinext/cloudflare` 1.0.0-beta.1
- `wrangler` 4.110.0
- Plain workers.dev URL — no custom domain, no route, no zone (so no cache purge available)
- App Router project, ISR home page with `export const revalidate = 300`
- Deploy command: `vinext-cloudflare deploy --config dist/server/wrangler.json --experimental-warm-cdn-cache`

## Expected behavior

My page sets `revalidate = 300`, which correctly gives the prerendered HTML a 300s edge TTL *within* a deployment. But a **new deployment should reset/invalidate that cache** (or the warmup should replace the entries with the new build). That's the entire promise of `--experimental-warm-cdn-cache`: after the deploy prints `1/1 path(s) warmed` and succeeds, the public URL should serve the new build.

## Actual behavior (reproduced twice)

Timeline from the second, cleaner replication (all times UTC; `V3` = old build, `V4` = new build, marker rendered in the page `

`):

| Time | Event |
|---|---|
| 01:30:13 | Pre-deploy: public URL serves `cf-cache-status: HIT`, `age: 90`, **V3** |
| 01:30:31 | Deploy starts (V4 build) |
| 01:30:34 | KV cache: uploaded 3 entries for 2 prerendered routes |
| 01:30:43 | New version `9dfc7739` staged at 0%; old `7d2677ad` still 100% |
| 01:30:44–45 | `Warming CDN cache for 1 build-discovered path(s)... CDN warmup: 1/1 path(s) warmed.` |
| 01:30:46 | `9dfc7739` promoted to 100%; deploy reports success |
| 01:30:46–47 | 8 immediate probes: **all return V3.** One probe returns `HIT, age: 2` — an entry created at 01:30:45, i.e. **by the warmup request itself, containing the old version's HTML** |
| 01:33:32 | Cache-busted request (`?bust=...`): `MISS` → **V4** (origin is fine). Plain URL: `HIT, age: 167` → **V3** — age dates this entry to exactly 01:30:45, the warmup second |
| 01:35:48 | `cf-cache-status: UPDATING`, `age: 302` → still **V3** |
| 01:36:03 | `HIT, age: 15` → **V4** finally public — **5m17s after deploy success** |

Note the poisoning effect: the pre-deploy entry (age 90 at 01:30:13) would have expired naturally at ~01:33:44. The entry the warmup created at 01:30:45 served the old build until ~01:35:48. **The flag extended the stale window by ~2 minutes compared to doing nothing.**

The first replication showed the same shape, plus a mixed-version window: within a single second the same URL returned the old build (`HIT, age: 48`) and the new build (`MISS`) depending on which edge cache node answered. For pages referencing hashed asset URLs this can pair old HTML with a new deployment's assets.

## Root cause

`deployWithCdnWarmup` in `@vinext/cloudflare/dist/deploy.js`:

1. Uploads the new version and stages it at **0%** traffic (old version stays at 100%).
2. Warms the **public production URL** while sending `Cloudflare-Workers-Version-Overrides: =""`, expecting the staged version to produce the response that gets cached.
3. Counts any response with status `< 400` as "warmed" (`warmOnePath` in `cdn-warm.js`), then promotes the new version to 100%.

The problem: **the version-override header is ignored on workers.dev.** Directly verifiable — after the deploy above, with `9dfc7739` (V4) at 100% and `7d2677ad` (V3) still resident:

```
$ curl -s -H 'Cloudflare-Workers-Version-Overrides: vinext-experiment="7d2677ad-3edf-4ec2-b8f3-dd11be67eb03"' \
"https://.workers.dev/?bust=1" | grep -o 'V[0-9]'
V4 # asked for the old version, got the new one → header ignored
```

So during warmup the request is routed by the traffic split — 100% to the **old** version — and one of two things happens on each edge cache node:

- The node already has a fresh entry → `HIT`, warmup "succeeds" by reading the stale cache and changes nothing.
- The node has no entry → `MISS`, the **old** version renders the response, and it is cached with a **full fresh TTL** under the public cache key. This is the poisoning case observed at 01:30:45.

Either way `warmOnePath` sees a `2xx` and reports the path as warmed. Nothing invalidates existing entries at promotion time, and on workers.dev there is no cache-purge API to fall back on.

Interestingly the code seems aware overrides need a proper zone — the `--warm-cdn-strict` error text says "pre-traffic warmup needs a production URL and Worker name for version overrides. Configure a route/custom domain..." — but in the default non-strict path, `resolveCdnWarmupTargetUrl` silently falls back to the workers.dev URL where the override doesn't work.

## Suggested fixes

Any of these would help, roughly in order of value:

1. **Verify the warmup response actually came from the staged version** before counting a path as warmed (e.g. echo the version/BUILD_ID in a response header and compare). This turns the silent failure into a visible one everywhere, not just workers.dev.
2. **Detect the workers.dev-only case** (no custom domain/route) and skip the pre-promotion warmup with a clear warning — falling through to the existing post-promotion warmup path — instead of poisoning the cache and reporting success.
3. Treat `cf-cache-status: HIT` on a warmup response as *not warmed* — a HIT means the warmup read an existing (old) entry rather than populating anything.
4. Document that on workers.dev, previously cached ISR pages (`revalidate = N`) keep serving the old build for up to N seconds after a deploy, since there's no purge and the warmup can't displace fresh entries.

## Repro steps

1. Create an App Router project with an ISR page (`export const revalidate = 300`) whose HTML contains a visible version marker; deploy to a plain workers.dev URL.
2. `curl` the public URL until `cf-cache-status: HIT` (fresh entry, marker = old version).
3. Change the marker, rebuild, and deploy with `--experimental-warm-cdn-cache`.
4. Immediately after the deploy prints `CDN warmup: 1/1 path(s) warmed.` and succeeds, `curl` the public URL repeatedly: it returns the old marker (`HIT`), and `Age` on the entries dates one of them to the exact second the warmup ran.
5. Optional: confirm the mechanism by sending `Cloudflare-Workers-Version-Overrides` pinning a non-live version with a cache-busting query — the header has no effect on workers.dev.

Contributor guide

Open the contributing guide

Research direction

Start in @vinext/cloudflare/dist/deploy.js at deployWithCdnWarmup and follow warmOnePath in cdn-warm.js, including resolveCdnWarmupTargetUrl. Reproduce the workers.dev deployment with an ISR version marker and curl probes, then verify the selected fix no longer reports stale responses as warmed or poisons the cache before promotion, while preserving the documented post-promotion behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.