Feature Request: Configurable partitioning for BigQuery exports
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 7h 27m
- Merged PRs (30d)
- 222
Description
Feature request
Is your feature request related to a problem?
Allow the granularity of timestamp partitioning for BigQuery exports to be configured, to help improve query performance.
Additional context
From: https://posthoghelp.zendesk.com/agent/tickets/27037 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/28726)
Seems we have a TODO to look at making partitioning configurable: https://github.com/PostHog/posthog/blob/78f4374769d53ddccc9a7eacb893fefe473c96a1/posthog/temporal/batch_exports/bigquery_batch_export.py#L186-L190
Debug info
Kind: support
Target area: apps
Report event: http://go/ticketByUUID/9e253d84-c05b-4000-8a7b-3b82f4cc84cf
Session: https://us.posthog.com/project/sTMFPsFhdP1Ssg/replay/019590f9-6cc4-7e16-ad7b-89ab97602c49?t=46
Exceptions: https://us.posthog.com/project/2/error_tracking?filterGroup=%7B%22type%22%3A%22AND%22%2C%22values%22%3A%5B%7B%22type%22%3A%22AND%22%2C%22values%22%3A%5B%7B%22key%22%3A%22%24session_id%22%2C%22value%22%3A%5B%22019590f9-6cc4-7e16-ad7b-89ab97602c49%22%5D%2C%22operator%22%3A%22exact%22%2C%22type%22%3A%22event%22%7D%5D%7D%5D%7D
Location: https://us.posthog.com/project/65144
Sentry: http://go/sentryUS/65144
Persons-on-events mode for project: person_id_override_properties_joined
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in posthog/temporal/batch_exports/bigquery_batch_export.py around lines 186-190, where the TODO about configurable partitioning is noted. Trace how BigQuery export settings are represented and passed into the exporter, then determine the supported timestamp partitioning options. Done means the partitioning granularity can be configured for BigQuery exports and the resulting export behavior is covered by the relevant tests, if present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100