databuddy-analytics / databuddy-analytics/Databuddy
perf(rpc): goals bulkAnalytics fires 2 ClickHouse queries per goal (N+1)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1.2k
- Forks
- 216
- Avg merge
- 14h 53m
- Merged PRs (30d)
- 154
Description
Problem
goalsRouter.bulkAnalytics (packages/rpc/src/routers/goals.ts) fires two ClickHouse queries per goal — one for the conversion count (processGoalConversionCount) and one for the total-website-users denominator (getTotalWebsiteUsers) — inside a Promise.all over every requested goal.
For a website with N goals, loading the goals dashboard issues up to 2×N concurrent ClickHouse round-trips on every request. A website with 20 goals fires ~40 queries where the actual underlying data (unique visitor counts scoped by date range) could largely be answered by a small, fixed number of queries.
Root cause
Each goal independently computes:
getTotalWebsiteUsers(websiteId, effectiveStartDate, endDate, combinedFilters)— the entrant denominatorprocessGoalConversionCount(step, combinedFilters, params)— the completion count for that goal's single step
Most goals on a given website share the same date range and carry no goal-specific filters, so steps 1 and 2 above are frequently computing the same denominator and issuing structurally identical completion queries that differ only in which single event/pageview they're matching — exactly the shape ClickHouse can answer in one query via GROUP BY.
Proposed fix
Group goals by (a) whether they carry any filters (request-level or goal-level) and (b) their effective start date:
- Goals with no filters at all in the same date bucket can be counted in a single batched query — one shared
getTotalWebsiteUserscall plus one query that matches all their step conditions and returns per-goal completion counts viaGROUP BY, instead of N×2 queries. - Goals with filters keep the existing one-query-per-goal path unchanged, since the shared event-stream query builder (
buildIdentifiedEventStream) only threads filter conditions through the step at array index 0 (correct for its actual funnel-entry-filter use case) — batching filtered goals together would silently apply another goal's filter to the wrong goal.
This bounds the fix to the case where it's provably correct without touching the shared funnel query builder, while still collapsing the common case (dashboards with mostly unfiltered goals) from O(N) queries to O(distinct date ranges).
Will open a PR shortly with the batched implementation, a regression test asserting the query count, and no change in output for existing behavior.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read packages/rpc/src/routers/goals.ts, focusing on goalsRouter.bulkAnalytics, processGoalConversionCount, and getTotalWebsiteUsers. Trace the existing Promise.all and ClickHouse query helpers first; done means unfiltered goals share batched counts by effective start-date bucket, filtered goals retain the existing path, and a regression test confirms query count and unchanged output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, data, databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 64/100