Team managed throttling of feature flag requests by distinct_id

Problem

A feature flags customer recently needed a small number of their users throttled because those users were generating disproportionate request volume. The customer identified them by distinct_id. We had no way to express that directly, so we resolved each distinct_id to a source IP and hardcoded the IP into Envoy's global rate limit configuration. This was done in charts#9712 (shadow) and charts#9720 (enforce).

This approach has several problems. We end up limiting IPs when the customer asked us to limit users, which is the wrong identifier because IPs rotate through NATs, mobile carriers, and shared CGNAT, so the rule hits bystanders and misses the real offenders. Envoy sees only an IP and a path, so the rule cannot be scoped to a single team. Every new entry requires a PostHog engineer to edit YAML in charts/ and wait on ArgoCD, which does not scale to many users or many customers. The rule tree on the shared ratelimit service also grows with each unrelated customer request.

Proposed solution

Give each team a first class list of distinct_ids to block or throttle at the feature flags service itself. Customers manage the list through the normal team settings API. The list is shipped to the Rust feature flags service via the same HyperCache mechanism already used to distribute flag definitions, so the hot path only does an in memory lookup.

High level design

1. Backend model. Add a TeamFeatureFlagRateLimitConfig Team extension (per the pattern in posthog/models/team/README.md) with three fields: a blocklist of distinct ids, an optional per id rate spec for soft throttling, and an enforcement mode (off, shadow, enforce). Expose the config on the existing Team API with admin level field access control.
2. Distribution. A Celery task pushes the config to a new HyperCache namespace keyed by team api token on every change, mirroring how flag_definitions_hypercache already works. Redis plus S3 backed, roughly 30 second propagation.
3. Enforcement. The Rust feature flags service loads each team's config from HyperCache into a HashSet (blocklist) and a HashMap of governor::Quota (rate limits). The check runs in handler::process_request immediately after cookieless::handle_distinct_id resolves the final identifier and before flag fetch and evaluation. Blocked requests return an empty flags response with a rate_limited=true marker so SDKs degrade gracefully. Throttled requests reuse the existing FlagsRateLimiter plumbing.
4. Shadow mode. shadow evaluates the rule and emits metrics and a canonical log field but still returns real flag results. Each team can validate its list before flipping to enforce, without a global environment variable toggle.
5. Observability. New metric flags_distinct_id_limit_evaluated_total{team_id, mode, action} for allow, shadow block, enforced block, shadow throttle, enforced throttle. Add a panel to feature-flags-general.json. The canonical log gains a distinct_id_rate_limited field so Loki queries can answer whether a specific request was blocked by a team rule.
6. Failure mode. If HyperCache is unavailable, fail open and allow the request, matching the existing billing limiter posture.

Performance

The hot path per request is one DashMap read on the HyperCache reader, one HashSet::contains, and optionally one governor::check_key if the id is also soft throttled. No Redis or Postgres call per request. Config refresh happens on a background interval.

Migration

Ship with mode=off as the default so no team is affected at deploy time. For the currently hardcoded customer, populate their config with the distinct ids that motivated the original IP rule, run in shadow for roughly 24 hours and compare against the Envoy counter in Grafana, then flip to enforce and revert the IP rule in charts.

Start with posthog/models/team/README.md and trace the existing flag_definitions_hypercache distribution flow. Then inspect handler::process_request, cookieless::handle_distinct_id, FlagsRateLimiter, and feature-flags-general.json to understand enforcement and observability entry points. Done means team-managed distinct_id blocklists and throttles support off, shadow, and enforce modes with fail-open behavior and the specified metrics and logs.