PostHog / PostHog/posthog

Feature Request: Configurable partitioning for BigQuery exports

Open
#29,932 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement team/batch-exports
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
7h 27m
Merged PRs (30d)
222

Description

Feature request

Is your feature request related to a problem?

Allow the granularity of timestamp partitioning for BigQuery exports to be configured, to help improve query performance.

Additional context

From: https://posthoghelp.zendesk.com/agent/tickets/27037 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/28726)

Seems we have a TODO to look at making partitioning configurable: https://github.com/PostHog/posthog/blob/78f4374769d53ddccc9a7eacb893fefe473c96a1/posthog/temporal/batch_exports/bigquery_batch_export.py#L186-L190

Debug info
Kind: support

Target area: apps

Report event: http://go/ticketByUUID/9e253d84-c05b-4000-8a7b-3b82f4cc84cf

Session: https://us.posthog.com/project/sTMFPsFhdP1Ssg/replay/019590f9-6cc4-7e16-ad7b-89ab97602c49?t=46

Exceptions: https://us.posthog.com/project/2/error_tracking?filterGroup=%7B%22type%22%3A%22AND%22%2C%22values%22%3A%5B%7B%22type%22%3A%22AND%22%2C%22values%22%3A%5B%7B%22key%22%3A%22%24session_id%22%2C%22value%22%3A%5B%22019590f9-6cc4-7e16-ad7b-89ab97602c49%22%5D%2C%22operator%22%3A%22exact%22%2C%22type%22%3A%22event%22%7D%5D%7D%5D%7D

Location: https://us.posthog.com/project/65144

Sentry: http://go/sentryUS/65144

Persons-on-events mode for project: person_id_override_properties_joined

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in posthog/temporal/batch_exports/bigquery_batch_export.py around lines 186-190, where the TODO about configurable partitioning is noted. Trace how BigQuery export settings are represented and passed into the exporter, then determine the supported timestamp partitioning options. Done means the partitioning granularity can be configured for BigQuery exports and the resulting export behavior is covered by the relevant tests, if present.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.