[Self-hosted] Inconsistent behavior when cleaning up duplicate events
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 7h 27m
- Merged PRs (30d)
- 222
Description
Bug Description
Sometimes duplicate events won't get cleared up, e.g.:
I also noticed some odd behavior. It seems weird, but it looks like:
- when there are only 2 duplicates, nothing happens
- when there are more than 2 duplicates, they get cleared up
For testing purposes I have configured the cron to be executed every minute:
CLEAR_CLICKHOUSE_REMOVED_DATA_SCHEDULE_CRON: '0 * * * *'
Could you please point me in the right direction where to look for errors?
Debug info
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
Latest build (same behavior was observed on the image 2 weeks old)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the cleanup behavior in a self-hosted Docker Compose deployment using the CLEAR_CLICKHOUSE_REMOVED_DATA_SCHEDULE_CRON setting. Compare runs with two duplicate events against runs with more than two, then trace the cleanup job and verify that duplicate events are consistently removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, docker-compose, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100