SetClientAssertionJWT inline DELETE on hydra_oauth2_jti_blacklist causes lock contention and latency spikes under concurrent load

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

No response

Describe the bug

Every call to /oauth2/token with grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer triggers an inline cleanup DELETE before inserting the new JTI:

// persistence/sql/persister_oauth2.go
if err := p.QueryWithNetwork(ctx).Where("expires_at < CURRENT_TIMESTAMP").Delete(&oauth2.BlacklistedJTI{}); err != nil {
    return sqlcon.HandleError(err)
}

Under concurrent load, multiple goroutines simultaneously issue this DELETE targeting the same expired rows. PostgreSQL acquires row-level locks on the matching rows; subsequent transactions block waiting for the first to commit. This manifests as latency spikes of 100–400ms on the /oauth2/token endpoint.

Observed behaviour

• p50/p95/p99 latencies are normal (~18ms / ~23ms / ~26ms)
• Isolated spikes of 100–400ms appear 1–3 times per hour under moderate load (~550 req/hour across 3 pods)
• APM traces show the spike time entirely within the DB span
• Database server metrics show no slow queries — the query executes in <1ms on the server side, but the application waits for the lock

The comment in the source (// this cleanup spares us the need for a background worker) suggests this is intentional, but the approach is unsafe under concurrency.

Root cause

The inline DELETE with no SKIP LOCKED means N concurrent requests all contend for locks on the same expired rows. The first acquires the lock and deletes; the rest wait. Wait time grows with row count and concurrency.

Suggested fixes

Option 1 (minimal): Use SKIP LOCKED to avoid blocking if another transaction already holds the lock on expired rows:

DELETE FROM hydra_oauth2_jti_blacklist
WHERE nid = ? AND expires_at < CURRENT_TIMESTAMP
  AND id IN (
    SELECT id FROM hydra_oauth2_jti_blacklist
    WHERE nid = ? AND expires_at < CURRENT_TIMESTAMP
    LIMIT 100
    FOR UPDATE SKIP LOCKED
  )

Option 2 (config flag): Add a config option to disable inline cleanup, delegating to hydra janitor. Users running a janitor CronJob are double-paying — the janitor runs AND every request also attempts cleanup.

Option 3 (async): Fire the cleanup in a goroutine rather than blocking the request path.

Workaround

Run hydra janitor --tokens on a frequent schedule (e.g. every 5 minutes) to keep the table near-empty, minimising the number of rows contested. This reduces spike frequency but does not eliminate it — lock contention can still occur between janitor runs.

Reproducing the bug

1. Deploy Hydra with grant_type=jwt-bearer and ttl.access_token=15m
2. Run 3+ replicas receiving concurrent token requests (~100+ req/min total)
3. Allow expired JTI rows to accumulate (no janitor running)
4. Observe latency spikes in APM correlated with DELETE FROM hydra_oauth2_jti_blacklist WHERE expires_at < CURRENT_TIMESTAMP

Relevant log output

Relevant configuration

Version

2.2.0 (observed with 26.2.0 as well)

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Kubernetes with Helm

Additional Context

No response

Start in persistence/sql/persister_oauth2.go at the inline expired-JTI DELETE used by the JWT bearer /oauth2/token path. Reproduce concurrent requests against PostgreSQL with expired rows and no janitor, then inspect lock waits and transaction behavior. Done means cleanup no longer causes token-request lock contention, with coverage for concurrent cleanup behavior.