`generate_demo_data` crashes with `BufferError: Local: Queue full`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Bug Description
generate_demo_data management command crashes with BufferError: Local: Queue full when producing simulated events to Kafka.
The _save_past_sim_events method in products/demo/backend/logic/matrix/manager.py writes all events for each simulated person in a tight loop via create_event() → KafkaProducer.produce(), without ever flushing the producer's internal buffer. When the number of events exceeds the librdkafka buffer capacity, the producer raises a BufferError.
How to reproduce
- Start the dev stack with
hogli start - The command crashes during demo data generation with:
BufferError: Local: Queue full
Additional context
- The Kafka broker is healthy
_save_past_sim_eventsiterates every event and callscreate_event()without any periodicflush()call on the Kafka producer- The producer does
poll(0)(non-blocking) after each message, but this only triggers delivery callbacks — it does not wait for the buffer to drain - A fix would be to periodically flush the producer (e.g. every N events) to let the buffer drain before producing more messages
Debug info
- [ ] PostHog Cloud, region and project ID: [please provide, you can find both at https://app.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in products/demo/backend/logic/matrix/manager.py at _save_past_sim_events, then trace create_event() to KafkaProducer.produce(). Check how the producer currently uses poll(0) and determine where periodic flushing belongs. Run the demo stack with hogli start and confirm generate_demo_data completes without BufferError: Local: Queue full.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, python
- Domain
- backend, stream-processing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100