cloudflare / cloudflare/workers-sdk
π BUG: concurrent processes race the single-use refresh token (read β exchange β write is unlocked); a lost race kills the refresh chain until interactive re-login
- Dominant language
- TypeScript
- Stars
- 4.5k
- Forks
- 1.5k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 187
Description
## Summary
`refreshToken()` still has a process-crossing race that #13910 did not close: the read β token-endpoint exchange β `writeFileSync` sequence on the shared auth config (`~/Library/Preferences/.wrangler/config/default.toml` on macOS) is not serialized or atomic. When two wrangler processes cross the access-token expiry boundary together, both read the same single-use refresh token and both send it to `https://dash.cloudflare.com/oauth2/token`. At most one exchange can win; on this machine the observed end state (twice in five days of logs) is a refresh chain that dies permanently β every subsequent refresh gets `400 Bad Request` (`invalid_grant`) until an interactive `wrangler login`.
#13910 fixed the *stale in-memory* variant (a long-lived process re-reads the rotated token from disk before exchanging β thank you, that fix is present and working in 4.113.0, where this was observed). What remains is the tight window *between* that fresh read and the write-back: it spans a network round-trip, so with enough concurrent invocations two processes land inside it.
## Environment
- wrangler 4.113.0 (bundled per-project, ~20 git worktrees of the same repo on one machine, all sharing the one global auth config)
- macOS (Darwin 25.5.0), Node 26
- Automation (a local dashboard + pipeline jobs) shells out to `wrangler d1 execute --remote` frequently: ~7,000 wrangler invocations/day visible in `~/Library/Preferences/.wrangler/logs/` (120,275 log files over 17 days)
- OAuth access-token lifetime observed: 1 hour β ~24 refresh windows/day, each a race opportunity
## Evidence from wrangler's own debug logs (timestamps only; tokens redacted)
Concurrent refreshes do land inside the window. Two invocations refreshed **4 ms apart** and both commands then errored:
```
wrangler-2026-08-03_01-58-29_679.log fetching auth token grant_type=refresh_token @ 01:58:29.935Z
wrangler-2026-08-03_01-58-29_682.log fetching auth token grant_type=refresh_token @ 01:58:29.931Z
(both processes then fail their `d1 execute` with APIError ~575 ms later)
```
The chain survived that one. It did not survive the next collision window:
```
2026-08-03T20:05:56Z last successful refresh (grant_type=refresh_token, no error)
2026-08-03T20:55:35Z refresh β "Failed to fetch auth token: 400 Bad Request"
2026-08-03T20:55:37Z refresh β 400 (second process, 2 s later)
... 40+ consecutive 400s across two worktrees through 2026-08-04T13:16Z,
all "Token refresh failed" at debug level, surfaced to the user as
"Your auth token has expired and could not be refreshed, and the
environment is non-interactive"
(machine idle Aug 5β6)
2026-08-07T23:11:27Z first refresh of the day β 400 immediately (chain still dead)
2026-08-07T23:17:15Z manual `wrangler login` β "Successfully logged in"
2026-08-08T04:05Z, 07:37Z healthy hourly refreshes resume
```
Once the stored refresh token is invalid, nothing self-heals: every process reads the same dead token, 400s, and (interactively) prompts a full OAuth login β which matches the widely-reported "wrangler makes me log in every day" experience for anyone running wrangler from more than one process.
## Where the race lives
`packages/workers-auth/src/core/file-storage.ts` β `write()` is a bare `writeFileSync` (no lock, no temp-file + atomic `rename`). `packages/workers-auth/src/flow.ts` `refreshToken()` does read-from-disk β `exchangeRefreshTokenForAccessToken` (network) β `storage.write(...)` with no mutual exclusion across processes. The refresh failure is also swallowed into `logger.debug`, so affected users never see the 400 without `WRANGLER_LOG=debug`.
## Suggested fix directions
1. **Advisory file lock around the whole refresh** (read β exchange β write), e.g. `proper-lockfile` on the config path; on lock acquisition, re-check `expiration_time` β if another process already refreshed while we waited, skip the exchange entirely. This closes the race for all wrangler processes on the machine, which is the common case.
2. Failing that, **atomic write** (temp file + `rename`) plus a **retry-on-`invalid_grant`**: re-read the config, and if the on-disk refresh token differs from the one just sent, retry with the new one before giving up.
3. Independently: log refresh-exchange 400s at warn level rather than debug β the silent failure made this take days to diagnose.
The author of #13910 noted a proper flock-style fix was "materially harder" and deferred it; this issue is the data point that the remaining window gets hit in practice.
Contributor guide
Assessment
This issue has not been assessed yet.