PostHog / PostHog/posthog

Switching a warehouse schema to webhook sync leaves a data gap - run a catch-up sync on method change

Open
#95,040 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement team/warehouse-sources
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Problem

When a schema's sync method changes to webhook, the switch leaves a data gap. The last batch sync ran at some point in the past. The webhook only captures changes from registration onward. Every row that changed between those two moments never reaches the table, and nothing in the switch path reconciles that window.

A user hit this on a Stripe customer schema that moved from full replica to webhook. They recovered by running one more full replica and then switching to webhook again.

The switch path only registers the consumer hog function — _maybe_create_webhook at products/warehouse_sources/backend/presentation/views/external_data_schema.py:1138, into products/data_warehouse/backend/logic/external_data_source/webhooks.py. It does not set reset_pipeline and does not trigger a refresh.

Describe the solution you'd like

When sync_type changes to webhook: register the webhook first, then run one catch-up sync with the previous method.

  • Cursor-based incremental catch-up where the table has an incremental field.
  • Full refresh only where it does not.

Overlap is safe, because rows merge by key. A gap is not.

Warehouse syncs bill per row, so this probably should not fire silently. An opt-in prompt at switch time ("Run a catch-up sync now?") may be the right shape. That part is a product call for the owning team.

Precedent already in the code

Two nearby paths do this kind of reconciliation:

  • Re-enabling a paused webhook schema (should_sync False → True) sets reset_pipeline and triggers a refresh, "so a gap from the off-window gets filled" (external_data_schema.py:1024).
  • A sync-type change into or out of XMIN forces reset_pipeline and a refresh (external_data_schema.py:1012).

A switch to webhook has the same gap property, but no equivalent handling.

Describe alternatives you've considered

  • Document the gap and leave the behavior as is. Cheapest, but users lose rows silently and find out much later.
  • Always force a full refresh on the switch. Correct, but expensive on large tables. The incremental path covers most cases for less cost.

Additional context

Reported by a customer in Slack: https://posthog.slack.com/archives/C0BMHNZT1NZ/p1788511818073059?thread_ts=1788511818.073059&cid=C0BMHNZT1NZ

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in products/warehouse_sources/backend/presentation/views/external_data_schema.py around lines 1012, 1024, and 1138, then trace the webhook registration in products/data_warehouse/backend/logic/external_data_source/webhooks.py. Compare the paused-webhook and XMIN reconciliation paths with the sync-type switch. Done means the switch path handles the gap and the catch-up cost or opt-in behavior is resolved with the owning team.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.