Visibility into what's in the ingestion queues and past performance

why: To help reduce time to incident resolution and help find bottlenecks

Plan:

• Fix partition stats
  1. Understand why current partition stats is broken sometimes, see if there is a quick fix:
     - delete table and re-create with a better schema?
     - keep time, topic, partition, token, event_name, distinct_id dimensions, count + size metric
     - 15 minutes granularity, partition by hour with expiration
     - one destination table, consuming all raw topics -> cannot monitor the sessions kafka cluster :sadtaco:
• Aggregated performance characteristics (per team, event type, distinct_id), for example:
  • How long certain steps (person, group, plugins) took
  • Hopefully runbooks can be super easy “check out this awesome table”

Additional info:
Q2 planning doc: https://docs.google.com/document/d/1pC9P5dRSh7NP3ActXcAe1YaN4Y0nUyEeJI3noM1KpFw/edit

• https://github.com/PostHog/posthog/issues/20517

From incidents follow-ups:

• visibility into the backlog for different ingestion topics (aka partition stats v2 - this is already part of Q2 plans, this increases in priority)
• visibility into what has been causing problems in the ingestion pipeline (e.g. slow sql query logs, extra column in events table with timing info)

No files, tests, or entry points are named. Start by reviewing linked issue #20517 and the Q2 planning document, then map the ingestion queues, partition stats, raw topics, and performance data described here. Done would require agreed implementation scope plus visibility into queue backlogs and ingestion performance.