Make the LLM gateway policy cache Redis-only, matching the gateway credential cache
Open
@haacked is already working on this.
Since Aug 10, 2026.
feature/ai-gateway
team/ai-gateway
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Problem
team_llm_gateway_policy_hypercache (posthog/storage/team_llm_gateway_policy_cache.py) keeps an S3 tier it doesn't need. The Go gateway reads the policy blob from Redis directly, and the only Django-side production reader is the admin page's Redis-only probe. The sibling gateway_credential_hypercache already documents why an auth-adjacent cache should skip S3:
Redis-only: a ~1h OAuth token would outlive an S3 lifecycle and resurrect on a cold Redis.
The same reasoning applies to the policy blob, with two concrete costs today:
- Stale resurrection. S3 objects don't expire with the 7-day Redis TTL, and the tiered read path repairs an S3 hit back into Redis. A cold or flushed Redis can resurrect a policy that rotation or revocation already cleared.
- Deletes depend on S3 availability. With hypercache deletes going S3-first (follow-up to #78070: the S3 delete now gates the Redis delete, so a partial failure can't strand a repairable orphan), the token-rotation and admission-change clears in
team_llm_gateway_policy_signal_handlers.pyraise on a transient S3 error before touching Redis. The surroundingexcept Exceptionswallows that, and nothing re-drives a rotated-away token's key: the hourly refresh resolves sorted-set members viaTeam.objects.filter(api_token__in=...), which no longer matches the old token. A Redis-only cache removes the S3 dependency from those clears entirely.
Proposal
- Pass
s3_enabled=Falsetoteam_llm_gateway_policy_hypercache. Writes, reads, and deletes then skip the S3 tier; expiry tracking and the hourly refresh are unaffected. - While in there, consider doing the same for
gateway_credential_hypercache. It is Redis-only by call-site convention today (set_cache_value_redis_only,kinds=["redis"]), but its constructor leavess3_enableddefaulted toTrue, so nothing structural stops a future writer path from populating S3 and reintroducing the resurrection risk its docstring warns about.
Considerations:
- Cold-Redis behavior: the gateway fails closed on a missing key and the hourly refresh re-projects entries; that's the same trade the gateway credential cache accepted.
- Audit
LLM_GATEWAY_POLICY_HYPERCACHE_MANAGEMENT_CONFIG, the verifier, and any warm jobs for S3 assumptions before flipping the flag. - Existing
llm_gateway_policy.jsonS3 objects become orphaned; delete them in a one-off pass or let the bucket lifecycle reap them.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.