PostHog / PostHog/posthog

`generate_demo_data` crashes with `BufferError: Local: Queue full`

Open Beginner friendly
#96,037 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Bug Description

generate_demo_data management command crashes with BufferError: Local: Queue full when producing simulated events to Kafka.

The _save_past_sim_events method in products/demo/backend/logic/matrix/manager.py writes all events for each simulated person in a tight loop via create_event()KafkaProducer.produce(), without ever flushing the producer's internal buffer. When the number of events exceeds the librdkafka buffer capacity, the producer raises a BufferError.

How to reproduce

  1. Start the dev stack with hogli start
  2. The command crashes during demo data generation with:
BufferError: Local: Queue full

Additional context

  • The Kafka broker is healthy
  • _save_past_sim_events iterates every event and calls create_event() without any periodic flush() call on the Kafka producer
  • The producer does poll(0) (non-blocking) after each message, but this only triggers delivery callbacks — it does not wait for the buffer to drain
  • A fix would be to periodically flush the producer (e.g. every N events) to let the buffer drain before producing more messages
Debug info
- [ ] PostHog Cloud, region and project ID: [please provide, you can find both at https://app.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in products/demo/backend/logic/matrix/manager.py at _save_past_sim_events, then trace create_event() to KafkaProducer.produce(). Check how the producer currently uses poll(0) and determine where periodic flushing belongs. Run the demo stack with hogli start and confirm generate_demo_data completes without BufferError: Local: Queue full.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, python
Domain
backend, stream-processing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.