koala73 / koala73/worldmonitor

fix(vercel-ignore): same-commit redeploy is always skipped, so env var changes cannot reach production

Open
#8,216 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Summary

An environment variable added or rotated in Vercel cannot reach production on `main` by redeploying, because `scripts/vercel-ignore.sh` cancels every same-commit redeploy. The standard recovery for a credential change — "set the value, then redeploy" — silently does nothing.

This extended the #8208 outage. Checkout, customer-portal and notification-channels stayed down for ~45 minutes *after* the correct secret was in Vercel, across two redeploy attempts.

## Reproduction

1. Set or rotate any env var in Vercel production.
2. Redeploy the current `main` commit (dashboard Redeploy, or `vercel --prod`).
3. The deployment is canceled after ~7s and production keeps serving the old build, which does not have the value.

```
Running "bash scripts/vercel-ignore.sh"
Skipping: no web-relevant changes on main
The deployment was canceled because the Ignored Build Step command returned exit code 0.
```

Observed on deployments `worldmonitor-5z2wvbamw` (02:29Z) and `worldmonitor-jofyfh6f2` (02:51Z) on 2026-09-16.

## Cause

`scripts/vercel-ignore.sh:6-20` decides on a path diff:

```bash
WEB_CHANGES=$(git diff --name-only "$VERCEL_GIT_PREVIOUS_SHA" HEAD -- 'src/' 'api/' ...)
[ -z "$WEB_CHANGES" ] && echo "Skipping: no web-relevant changes on main" && exit 0
```

On a same-commit redeploy `VERCEL_GIT_PREVIOUS_SHA` is `HEAD`, so the diff is necessarily empty and the skip is unconditional. The check is correct for its actual purpose (don't rebuild for a docs-only merge) — it just has no notion of "the build inputs changed even though the code did not".

Two things make it expensive to diagnose:

- The deployment reports as **`Canceled`**, which reads like a transient infra failure rather than a deliberate skip. Nothing distinguishes it from a real cancellation in `vercel ls`.
- Vercel binds env vars at **build** time, so the symptom is a config-missing branch (`api/create-checkout.ts:164` → 503) in code that looks correctly written.

## Proposed fix

Options, roughly in order of preference:

1. **Escape hatch in the script.** Honour an explicit override so a redeploy can force a build — e.g. build unconditionally when `VERCEL_GIT_PREVIOUS_SHA == HEAD` (a same-commit redeploy is by definition a deliberate manual action, never a merge-train event). This makes the dashboard button work as everyone expects.
2. **Document the real procedure** in `docs/relay-credentials.md` and anywhere else that says "set the variable and redeploy": the working command is `vercel --prod --force --scope eliewm` (`--force` bypasses the Ignored Build Step).
3. Make the skip legible — the current message says *why* it skipped but not that env changes will not land.

(1) and (2) are not exclusive; (2) is worth doing regardless since it is the recovery path under incident pressure.

## Diagnostic worth keeping

Variable newer than the newest READY production build ⇒ production does not have it:

```
vercel env ls production --scope eliewm --project worldmonitor # prints each var's age
vercel ls worldmonitor --scope eliewm --prod # prints each deployment's age + status
```

Credential-free liveness probe for the gateway env (returns `misconfigured` if any required var is missing in the serving build, `invalid_state` if all are present):

```
curl -A "worldmonitor-ops-diagnostic/1.0" \
"https://www.worldmonitor.app/api/discord/oauth/callback?code=probe&state=nonexistent"
```

Note the project lives under the `eliewm` team and the repo has no `.vercel/` link, so both CLI commands need explicit `--scope`/`--project`.

Context: #8208, and the validation comment https://github.com/koala73/worldmonitor/issues/8208#issuecomment-5691556926

Contributor guide

Open the contributing guide

Research direction

Start with scripts/vercel-ignore.sh:6-20 and review how Vercel sets VERCEL_GIT_PREVIOUS_SHA for same-commit redeploys. Check docs/relay-credentials.md for the current recovery procedure, then validate the chosen behavior with a forced or same-commit deployment; done means environment-variable changes can reach production and the documented command works.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash
Domain
build-system, cloud, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.