Warehouse sync from a Postgres database is rough
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Feature request
Is your feature request related to a problem?
We're having issues with syncing data from our postgres database into Posthog. The vast majority of the time, the sync goes smoothly and the data makes it in. However, we get occasional timeouts every ~20 days or so on random tables.
When this happens, Postgres automatically disables the sync for that table going forward. The only way that we notice that things are broken is when one of our dashboard queries starts looking to be nonsense, in which case we go in, see the table sync is in a failed state, and reenable it.
For example:
Describe the solution you'd like
Ideally:
- When a table fails to sync, it automatically retries with exponential backoff
- There should be some sort of a configurable notification if a sync fails
Describe alternatives you've considered
?? Not much. Come back and check the dashboard periodically?
Additional context
Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue describes a Postgres-to-PostHog warehouse sync but names no repository files, tests, or entry points. First locate the sync implementation and its failure handling, then clarify the retry/backoff behavior and configurable notification requirements. Done means failed table syncs recover automatically and surface notifications as specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100