Make cohort processor disaster recovery real before downstream depends on membership state
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
All membership evaluation state lives in RocksDB on a single pod's local volume. Losing that volume today means rerunning backfills, which is annoying but harmless because nothing consumes the output yet. Once the cohort_membership Postgres table feeds workflows or feature flags, a lost volume becomes a long outage while state rebuilds.
The processor has an S3 checkpoint and restore layer, but it is off by default (CHECKPOINT_ENABLED in rust/cohort-stream-processor/src/config.rs), has never run in production, and no bucket or access role is provisioned.
Known defects to fix before enabling it. There is no periodic full upload, so a checkpoint can reference arbitrarily old files and a bucket retention policy would delete files a restore still needs, silently breaking recovery. A failed restore logs a warning and cold starts with an empty store instead of alerting. Enabling checkpoints without configuring a bucket silently does nothing. There is no metric for bytes uploaded, so cost cannot be sized.
Fix the defects, provision the bucket and access, then run a restore drill and verify that a pod restored from S3 serves correct membership.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/cohort-stream-processor/src/config.rs and trace the S3 checkpoint and restore entry points, including how restore failures and missing bucket configuration are handled. Check the existing RocksDB state flow and metrics setup before making changes. Done means checkpoint retention is safe, failures alert, upload bytes are measurable, access is provisioned, and a restore drill verifies correct membership from S3.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, postgres, rust
- Domain
- cloud, databases, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100