PostHog / PostHog/posthog

Make cohort processor disaster recovery real before downstream depends on membership state

Open
#88,015 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature/cohorts feature/feature-flags team/feature-flags
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

All membership evaluation state lives in RocksDB on a single pod's local volume. Losing that volume today means rerunning backfills, which is annoying but harmless because nothing consumes the output yet. Once the cohort_membership Postgres table feeds workflows or feature flags, a lost volume becomes a long outage while state rebuilds.

The processor has an S3 checkpoint and restore layer, but it is off by default (CHECKPOINT_ENABLED in rust/cohort-stream-processor/src/config.rs), has never run in production, and no bucket or access role is provisioned.

Known defects to fix before enabling it. There is no periodic full upload, so a checkpoint can reference arbitrarily old files and a bucket retention policy would delete files a restore still needs, silently breaking recovery. A failed restore logs a warning and cold starts with an empty store instead of alerting. Enabling checkpoints without configuring a bucket silently does nothing. There is no metric for bytes uploaded, so cost cannot be sized.

Fix the defects, provision the bucket and access, then run a restore drill and verify that a pod restored from S3 serves correct membership.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/cohort-stream-processor/src/config.rs and trace the S3 checkpoint and restore entry points, including how restore failures and missing bucket configuration are handled. Check the existing RocksDB state flow and metrics setup before making changes. Done means checkpoint retention is safe, failures alert, upload bytes are measurable, access is provisioned, and a restore drill verifies correct membership from S3.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, postgres, rust
Domain
cloud, databases, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.