cloudflare / cloudflare/workers-sdk

πŸ› BUG: concurrent processes race the single-use refresh token (read β†’ exchange β†’ write is unlocked); a lost race kills the refresh chain until interactive re-login

Open
#15,089 2 comments 0 reactions 1 assignee Claimed by @dario-piotrowicz View on GitHub
Dominant language
TypeScript
Stars
4.5k
Forks
1.5k
Avg merge
3d 8h
Merged PRs (30d)
187

Description

## Summary

`refreshToken()` still has a process-crossing race that #13910 did not close: the read β†’ token-endpoint exchange β†’ `writeFileSync` sequence on the shared auth config (`~/Library/Preferences/.wrangler/config/default.toml` on macOS) is not serialized or atomic. When two wrangler processes cross the access-token expiry boundary together, both read the same single-use refresh token and both send it to `https://dash.cloudflare.com/oauth2/token`. At most one exchange can win; on this machine the observed end state (twice in five days of logs) is a refresh chain that dies permanently β€” every subsequent refresh gets `400 Bad Request` (`invalid_grant`) until an interactive `wrangler login`.

#13910 fixed the *stale in-memory* variant (a long-lived process re-reads the rotated token from disk before exchanging β€” thank you, that fix is present and working in 4.113.0, where this was observed). What remains is the tight window *between* that fresh read and the write-back: it spans a network round-trip, so with enough concurrent invocations two processes land inside it.

## Environment

- wrangler 4.113.0 (bundled per-project, ~20 git worktrees of the same repo on one machine, all sharing the one global auth config)
- macOS (Darwin 25.5.0), Node 26
- Automation (a local dashboard + pipeline jobs) shells out to `wrangler d1 execute --remote` frequently: ~7,000 wrangler invocations/day visible in `~/Library/Preferences/.wrangler/logs/` (120,275 log files over 17 days)
- OAuth access-token lifetime observed: 1 hour β†’ ~24 refresh windows/day, each a race opportunity

## Evidence from wrangler's own debug logs (timestamps only; tokens redacted)

Concurrent refreshes do land inside the window. Two invocations refreshed **4 ms apart** and both commands then errored:

```
wrangler-2026-08-03_01-58-29_679.log fetching auth token grant_type=refresh_token @ 01:58:29.935Z
wrangler-2026-08-03_01-58-29_682.log fetching auth token grant_type=refresh_token @ 01:58:29.931Z
(both processes then fail their `d1 execute` with APIError ~575 ms later)
```

The chain survived that one. It did not survive the next collision window:

```
2026-08-03T20:05:56Z last successful refresh (grant_type=refresh_token, no error)
2026-08-03T20:55:35Z refresh β†’ "Failed to fetch auth token: 400 Bad Request"
2026-08-03T20:55:37Z refresh β†’ 400 (second process, 2 s later)
... 40+ consecutive 400s across two worktrees through 2026-08-04T13:16Z,
all "Token refresh failed" at debug level, surfaced to the user as
"Your auth token has expired and could not be refreshed, and the
environment is non-interactive"
(machine idle Aug 5–6)
2026-08-07T23:11:27Z first refresh of the day β†’ 400 immediately (chain still dead)
2026-08-07T23:17:15Z manual `wrangler login` β†’ "Successfully logged in"
2026-08-08T04:05Z, 07:37Z healthy hourly refreshes resume
```

Once the stored refresh token is invalid, nothing self-heals: every process reads the same dead token, 400s, and (interactively) prompts a full OAuth login β€” which matches the widely-reported "wrangler makes me log in every day" experience for anyone running wrangler from more than one process.

## Where the race lives

`packages/workers-auth/src/core/file-storage.ts` β€” `write()` is a bare `writeFileSync` (no lock, no temp-file + atomic `rename`). `packages/workers-auth/src/flow.ts` `refreshToken()` does read-from-disk β†’ `exchangeRefreshTokenForAccessToken` (network) β†’ `storage.write(...)` with no mutual exclusion across processes. The refresh failure is also swallowed into `logger.debug`, so affected users never see the 400 without `WRANGLER_LOG=debug`.

## Suggested fix directions

1. **Advisory file lock around the whole refresh** (read β†’ exchange β†’ write), e.g. `proper-lockfile` on the config path; on lock acquisition, re-check `expiration_time` β€” if another process already refreshed while we waited, skip the exchange entirely. This closes the race for all wrangler processes on the machine, which is the common case.
2. Failing that, **atomic write** (temp file + `rename`) plus a **retry-on-`invalid_grant`**: re-read the config, and if the on-disk refresh token differs from the one just sent, retry with the new one before giving up.
3. Independently: log refresh-exchange 400s at warn level rather than debug β€” the silent failure made this take days to diagnose.

The author of #13910 noted a proper flock-style fix was "materially harder" and deferred it; this issue is the data point that the remaining window gets hit in practice.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.