PostHog / PostHog/posthog

Warehouse sync from a Postgres database is rough

Open
#47,412 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Feature request

Is your feature request related to a problem?

We're having issues with syncing data from our postgres database into Posthog. The vast majority of the time, the sync goes smoothly and the data makes it in. However, we get occasional timeouts every ~20 days or so on random tables.

When this happens, Postgres automatically disables the sync for that table going forward. The only way that we notice that things are broken is when one of our dashboard queries starts looking to be nonsense, in which case we go in, see the table sync is in a failed state, and reenable it.

For example:

Image

Describe the solution you'd like

Ideally:

  • When a table fails to sync, it automatically retries with exponential backoff
  • There should be some sort of a configurable notification if a sync fails

Describe alternatives you've considered

?? Not much. Come back and check the dashboard periodically?

Additional context

Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue describes a Postgres-to-PostHog warehouse sync but names no repository files, tests, or entry points. First locate the sync implementation and its failure handling, then clarify the retry/backoff behavior and configurable notification requirements. Done means failed table syncs recover automatically and surface notifications as specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.