apiKey corrupted by routine update from instance with different PAYLOAD_SECRET

Describe the Bug

Two Payload instances sharing one MongoDB but holding different PAYLOAD_SECRET values silently corrupt users.apiKey and users.apiKeyIndex on any payload.update to a user document — even when the data patch does not include apiKey.

The corruption chain:

1. afterRead.decryptKey (packages/payload/src/auth/baseFields/apiKey.ts) decrypts the stored ciphertext under the current instance's normalized secret. Wrong secret → silent garbage (auth/crypto.ts uses AES-256-CTR, which has no MAC).
2. The merged data passed to field hooks contains data.apiKey = <garbage>.
3. apiKeyIndex.beforeValidate sees data.apiKey truthy → HMAC(garbage, current_secret) overwrites the stored apiKeyIndex.
4. apiKey.beforeChange.encryptKey receives value = <garbage> → re-encrypts under the current secret → overwrites the stored ciphertext.

Net effect: the original plaintext API key is permanently invalidated. There is no error, no log line, and no warning. A user's API key works until the day a different-secret instance happens to touch their record.

Link to the code that reproduces this issue

https://github.com/ShaunMWallace/payload-secret-repro

Reproduction Steps

git clone https://github.com/ShaunMWallace/payload-secret-repro
cd payload-secret-repro
pnpm install
pnpm all

pnpm all runs four steps in sequence (each in its own Node process):

1. step1:setup — boots Payload with SECRET_A, creates a user, generates an API key K1 (encrypted under SECRET_A), persists {userId, K1} for downstream steps.
2. step2:auth-on-a-baseline — boots Payload with SECRET_A, looks up the user via apiKeyIndex = HMAC(K1, A). Expected: lookup succeeds.
3. step3:corrupt-via-b — boots Payload with SECRET_B (same MongoDB, different secret), runs:

   await payloadB.update({
     collection: 'users',
     id: state.userId,
     data: { name: 'edited-on-b' },   // NOTE: apiKey is intentionally NOT in data
   });
   

4. step4:auth-on-a-after — boots Payload with SECRET_A, looks up the user via apiKeyIndex = HMAC(K1, A). Expected: lookup fails.

Sample output (sanitized):

[step1] user id      : 69f3575a69710e191b9a2d8f
[step1] plaintext key: 6614fc54-8c26-4903-ab56-6ca7842a4d8c

[step2] auth lookup       : OK — found user 69f3575a69710e191b9a2d8f

[step3] cipher BEFORE B   : 7ed2b5a48d65cace5aa35921d7dd294a849907b2665c0536…
[step3] cipher AFTER  B   : 087f6229f337237c2182ca3e58f37c1995e7771de298389f…
[step3] cipher changed    : YES (B re-encrypted under SECRET_B)

[step4] auth lookup       : FAILED (expected) — apiKeyIndex no longer matches; original apiKey is now unrecoverable

The repository's README walks through the chain in detail and includes a supplementary script (pnpm investigate) that contrasts safe same-secret re-saves (cipher rotates harmlessly under same secret, index unchanged) vs unsafe different-secret re-saves (both fields clobbered), ruling out simple IV rotation as the underlying cause.

Which area(s) are affected?

• area: core
• db: mongodb

Environment Info

payload:                 3.84.1
@payloadcms/db-mongodb:  3.84.1
mongoose:                9.6.1
node:                    22
mongodb:                 7.x
OS:                      darwin (arm64)

Real-world impact

We hit this in a Next.js + Payload + Vercel production application after a forced PAYLOAD_SECRET rotation. Vercel captures env at build time, so long-lived branch deployments continued serving traffic with the previous PAYLOAD_SECRET baked in. Routine editorial saves on user records from those stale-secret deployments corrupted production API keys at random — with no detectable signal on either side.

The repro deliberately removes Vercel and Next.js from the picture (two local Node processes against one local MongoDB) to prove the bug lives entirely in @payloadcms/payload.

Suggested fixes (in order of preference)

1. Detect wrong-secret decryption. Switch auth/crypto.ts from AES-256-CTR to an authenticated mode (AES-256-GCM, or HMAC-over-CTR) so req.payload.decrypt throws on a wrong secret instead of returning garbage. Single biggest impact — would have surfaced our incident immediately rather than over weeks.
2. Skip the apiKey round-trip on updates that don't include it. If data.apiKey is absent from the user-supplied patch, leave the stored ciphertext + apiKeyIndex untouched. Makes routine updates safe across instances regardless of secret state.
3. Document the single-secret-per-database invariant. The current docs do not state that PAYLOAD_SECRET is a database-level invariant. Optionally surface a startup warning that samples a few existing apiKey ciphertexts and verifies they decrypt cleanly under the current secret.
4. Support graceful rotation. Allow PAYLOAD_SECRET to be a list ordered newest-first; decrypt against any, encrypt only against the first. Permits a phased rollout across multi-instance topologies without the all-or-nothing cliff.