PostHog / PostHog/posthog

duckgres sink: re-prime on table_suffix change (NEEDS_RESYNC on schema drift)

Open
#65,331 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Background

The v3 duckgres data-import sink resolves a team's target schema from DuckLakeBackfill.table_suffix (PR #65323): posthog_data_imports_<suffix>, falling back to posthog_data_imports_team_<id> when unset. team_id was immutable; a suffix is not.

Problem

If a team sets or changes table_suffix after the sink has already written its tables, _duckgres_schema_name starts resolving to a new schema. The sink would silently begin writing into the new (empty) schema, orphaning the fully-primed old one — a silent, partial-data outcome for that team's warehouse.

This is the exact "a suffix could be set but ignored" hazard the duckling DAG stack calls out, in the sink's shape.

Proposed fix (depends on #63144 backfill state machine + #65323 resolver)

  1. Record what was primed. Add primed_schema (CharField) to DuckgresSinkSchemaState; set it at mark_primed / the reconciler's PRIMED transition to the resolved duckgres_data_imports_schema(team_id) at prime time.
  2. Detect drift in the reconciler. When a PRIMED schema's current resolved schema != primed_schema, CAS-transition it to NEEDS_RESYNC (state already exists) so the next backfill re-primes into the new schema. Log + Sentry on the transition.
  3. Re-prime targets the new schema — falls out of the existing backfill flow once the state flips, since the backfill processor also resolves via _duckgres_schema_name.
  4. Old schema is left in place (not auto-dropped) — cleanup belongs to the separate deletion-lifecycle sweeper follow-up.

Acceptance criteria / tests

  • Setting a suffix on an already-PRIMED team → reconciler flips it to NEEDS_RESYNC; a fresh backfill primes posthog_data_imports_<suffix>; live batches then apply there.
  • Unset→unset and unchanged-suffix cases are no-ops (no spurious re-prime).
  • DB-backed test on the state transition.

Dependencies / ordering

  • Requires #65323 (resolver) and #63144 (backfill state machine + DuckgresSinkSchemaState) merged first.
  • Migration number for the primed_schema field must be assigned at implementation time (rebase onto current master head).
  • Per-org suffix uniqueness (enforced by the provision/enable flow #64939) is assumed; without it two teams can collide on posthog_data_imports_<suffix>.

Interim safety (until this lands)

After #65323 deploys, audit sink-enabled teams for a non-NULL table_suffix — any that already have one will move schemas and need a one-time manual re-prime (reset_duckgres_failed_runs --replan-backfill once #63144 is in).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with DuckgresSinkSchemaState, mark_primed, the reconciler's PRIMED transition, and _duckgres_schema_name, then trace the existing NEEDS_RESYNC state and backfill flow. Add the primed schema recording and drift transition, including logging and Sentry, and verify the DB-backed transition tests cover changed, unset, and unchanged suffixes plus re-priming into the new schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.