Switching a warehouse schema to webhook sync leaves a data gap - run a catch-up sync on method change
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Problem
When a schema's sync method changes to webhook, the switch leaves a data gap. The last batch sync ran at some point in the past. The webhook only captures changes from registration onward. Every row that changed between those two moments never reaches the table, and nothing in the switch path reconciles that window.
A user hit this on a Stripe customer schema that moved from full replica to webhook. They recovered by running one more full replica and then switching to webhook again.
The switch path only registers the consumer hog function — _maybe_create_webhook at products/warehouse_sources/backend/presentation/views/external_data_schema.py:1138, into products/data_warehouse/backend/logic/external_data_source/webhooks.py. It does not set reset_pipeline and does not trigger a refresh.
Describe the solution you'd like
When sync_type changes to webhook: register the webhook first, then run one catch-up sync with the previous method.
- Cursor-based incremental catch-up where the table has an incremental field.
- Full refresh only where it does not.
Overlap is safe, because rows merge by key. A gap is not.
Warehouse syncs bill per row, so this probably should not fire silently. An opt-in prompt at switch time ("Run a catch-up sync now?") may be the right shape. That part is a product call for the owning team.
Precedent already in the code
Two nearby paths do this kind of reconciliation:
- Re-enabling a paused webhook schema (
should_syncFalse → True) setsreset_pipelineand triggers a refresh, "so a gap from the off-window gets filled" (external_data_schema.py:1024). - A sync-type change into or out of XMIN forces
reset_pipelineand a refresh (external_data_schema.py:1012).
A switch to webhook has the same gap property, but no equivalent handling.
Describe alternatives you've considered
- Document the gap and leave the behavior as is. Cheapest, but users lose rows silently and find out much later.
- Always force a full refresh on the switch. Correct, but expensive on large tables. The incremental path covers most cases for less cost.
Additional context
Reported by a customer in Slack: https://posthog.slack.com/archives/C0BMHNZT1NZ/p1788511818073059?thread_ts=1788511818.073059&cid=C0BMHNZT1NZ
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in products/warehouse_sources/backend/presentation/views/external_data_schema.py around lines 1012, 1024, and 1138, then trace the webhook registration in products/data_warehouse/backend/logic/external_data_source/webhooks.py. Compare the paused-webhook and XMIN reconciliation paths with the sync-type switch. Done means the switch path handles the gap and the catch-up cost or opt-in behavior is resolved with the owning team.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100