PostHog / PostHog/posthog

Make the LLM gateway policy cache Redis-only, matching the gateway credential cache

Open
#80,796 0 comments 0 reactions 1 assignee View on GitHub

@haacked is already working on this.

Since Aug 10, 2026.

feature/ai-gateway team/ai-gateway
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Problem

team_llm_gateway_policy_hypercache (posthog/storage/team_llm_gateway_policy_cache.py) keeps an S3 tier it doesn't need. The Go gateway reads the policy blob from Redis directly, and the only Django-side production reader is the admin page's Redis-only probe. The sibling gateway_credential_hypercache already documents why an auth-adjacent cache should skip S3:

Redis-only: a ~1h OAuth token would outlive an S3 lifecycle and resurrect on a cold Redis.

The same reasoning applies to the policy blob, with two concrete costs today:

  • Stale resurrection. S3 objects don't expire with the 7-day Redis TTL, and the tiered read path repairs an S3 hit back into Redis. A cold or flushed Redis can resurrect a policy that rotation or revocation already cleared.
  • Deletes depend on S3 availability. With hypercache deletes going S3-first (follow-up to #78070: the S3 delete now gates the Redis delete, so a partial failure can't strand a repairable orphan), the token-rotation and admission-change clears in team_llm_gateway_policy_signal_handlers.py raise on a transient S3 error before touching Redis. The surrounding except Exception swallows that, and nothing re-drives a rotated-away token's key: the hourly refresh resolves sorted-set members via Team.objects.filter(api_token__in=...), which no longer matches the old token. A Redis-only cache removes the S3 dependency from those clears entirely.

Proposal

  • Pass s3_enabled=False to team_llm_gateway_policy_hypercache. Writes, reads, and deletes then skip the S3 tier; expiry tracking and the hourly refresh are unaffected.
  • While in there, consider doing the same for gateway_credential_hypercache. It is Redis-only by call-site convention today (set_cache_value_redis_only, kinds=["redis"]), but its constructor leaves s3_enabled defaulted to True, so nothing structural stops a future writer path from populating S3 and reintroducing the resurrection risk its docstring warns about.

Considerations:

  • Cold-Redis behavior: the gateway fails closed on a missing key and the hourly refresh re-projects entries; that's the same trade the gateway credential cache accepted.
  • Audit LLM_GATEWAY_POLICY_HYPERCACHE_MANAGEMENT_CONFIG, the verifier, and any warm jobs for S3 assumptions before flipping the flag.
  • Existing llm_gateway_policy.json S3 objects become orphaned; delete them in a one-off pass or let the bucket lifecycle reap them.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.