OAuth token exchange 500 when API_TOKEN_UPDATE outbox fails
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 21h 10m
- Merged PRs (30d)
- 635
Description
## Summary
`POST /oauth/token/` intermittently returns a 500 to OAuth clients (confirmed via MCP server — [MCP-SERVER-FQD](), 6,254 events, 4,097 affected users, ongoing as of 2026-07-08). The failure is caused by an `OutboxFlushError` on the `API_TOKEN_UPDATE` outbox shard (category 32) propagating to the HTTP response during a code exchange, not a true token creation failure.
## Root cause chain
1. `POST /oauth/token/` → `oauth_token.py::get_access_tokens` → `ApiToken.from_grant()` creates token
2. On transaction commit, outbox drains `API_TOKEN_UPDATE` shard (category 32) synchronously
3. `ApiToken.handle_async_replication` fires RPC: `region_replica.upsert_replicated_api_token`
4. Region returns `Invalid service request` → wrapped as `RpcRemoteException`
5. `OutboxFlushError: Could not flush shard category=32 (API_TOKEN_UPDATE)` propagates → 500
Sentry issues:
* [SENTRY-5QGA]() — 23,993 events on `/oauth/token/`, `upsert_replicated_api_token: Invalid service request`, first seen 2026-06-08
* [SENTRY-5QGB]() — 8,916 events on `/api/0/api-tokens/`, `region_tombstone.record_remote_tombstone: Invalid service request`, same category/date (tombstone path)
**Timeline note:** MCP-side failures started 2026-04-11 (predating these outbox issues). Current July 8 failures need correlation to confirm whether SENTRY-5QGA is still responsible or if there's an additional cause.
## Likely cause of "Invalid service request"
The rejection is probably a **contract/data validation failure** on the region side, not a transient availability issue. Likely candidates:
* RPC payload shape mismatch between control and region (field added/removed/changed)
* `RpcApiToken` contains a nullable or unexpected field region validation rejects
* Missing or invalid `organization_id`, `scoping_organization_id`, or `application_id`
* OAuth-created token shape rejected by region-side validation
* Version skew between control and region at deployment around 2026-06-08
To confirm: inspect the **region-side exception/log** for the same trace to get the real validation error behind "Invalid service request."
## Scope
Both SENTRY-5QGA and SENTRY-5QGB are category 32 (`API_TOKEN_UPDATE`), same start date, same "Invalid service request" failure:
* 5QGA: new token creation path (`upsert_replicated_api_token`)
* 5QGB: tombstone replication path (`record_remote_tombstone`)
## Fix direction
Two layers to address:
**Primary:** Fix the `region_replica.upsert_replicated_api_token` / `region_tombstone.record_remote_tombstone` rejection. Identify what "Invalid service request" means on the region side and correct the RPC contract or payload.
**Secondary:** Decide whether `OutboxFlushError` on the OAuth token path should cause a user-facing 500. If token replication is eventually consistent and retryable, the OAuth flow should not fail solely because a sync outbox delivery attempt was rejected — but this depends on whether immediate region availability is a correctness requirement.
## Related
* getsentry/sentry#118610 — Similar outbox-caused 500 on `/oauth/token/` but different failure: `InterfaceError` during `process_shard` cleanup on token **refresh**. Not a duplicate.
Contributor guide
Research direction
Start with oauth_token.py::get_access_tokens, ApiToken.from_grant(), and ApiToken.handle_async_replication, then correlate the failing trace with region-side logs for upsert_replicated_api_token and record_remote_tombstone. Determine what causes the region's "Invalid service request" and whether the outbox failure should be allowed to reach the OAuth response. Done means the responsible contract or payload issue and the appropriate OAuth error-handling decision are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, authentication, backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100