PostHog / PostHog/posthog

Move the CDP cohort membership consumer to the processor's output topic safely

Open
#88,010 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature/cohorts feature/feature-flags team/feature-flags
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

The CDP cohort membership consumer (CdpCohortMembershipConsumer, plugin server mode cdp-cohort-membership) upserts membership changes into the cohort_membership Postgres table. Its brokers still point at the cluster that hosts the legacy cohort_membership_changed topic, which no longer receives messages.

The move follows the topic creation in #88009. The consumer hardcodes the cohort_membership_changed topic name, so once the new topic exists on the cluster the cohort stream processor produces to, the move is a broker configuration change on the deployment alone. No consumer code changes are needed and #87866 is not needed. The consumer can move before or after the processor flips its writes, since reading an empty topic is harmless, but both must happen before anything depends on the Postgres table.

Three things need care during the move.

Keep the consumer group id unchanged. Autoscaling on consumer lag matches the literal group id, so changing it silently degrades scaling. The group has no committed offsets on the new cluster either way, so it starts fresh regardless.

Do not expect the table to converge on its own. The new topic starts empty, so there is no replay burst and no history to catch up from. The cohort_membership table converges through reconcile runs dispatched per cohort after the consumer is live, each of which replays that cohort's full current membership.

Add consumer lag alerting for this group. Topic settings are chosen deliberately at creation in #88009, but the consumer group needs lag alerting before the Postgres table depends on the feed, since lag past retention is silent data loss.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the deployment configuration for CdpCohortMembershipConsumer in plugin server mode cdp-cohort-membership, then inspect the consumer lag alerting configuration. Move the brokers to the processor's output cluster without changing the literal consumer group id, add lag alerting, and verify that reconcile runs can repopulate cohort_membership from the new topic.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, postgresql
Domain
data-engineering, databases, devops, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.