databuddy-analytics / databuddy-analytics/Databuddy

perf(rpc): goals bulkAnalytics fires 2 ClickHouse queries per goal (N+1)

Open
#679 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1.2k
Forks
216
Avg merge
14h 53m
Merged PRs (30d)
154

Description

Problem

goalsRouter.bulkAnalytics (packages/rpc/src/routers/goals.ts) fires two ClickHouse queries per goal — one for the conversion count (processGoalConversionCount) and one for the total-website-users denominator (getTotalWebsiteUsers) — inside a Promise.all over every requested goal.

For a website with N goals, loading the goals dashboard issues up to 2×N concurrent ClickHouse round-trips on every request. A website with 20 goals fires ~40 queries where the actual underlying data (unique visitor counts scoped by date range) could largely be answered by a small, fixed number of queries.

Root cause

Each goal independently computes:

  1. getTotalWebsiteUsers(websiteId, effectiveStartDate, endDate, combinedFilters) — the entrant denominator
  2. processGoalConversionCount(step, combinedFilters, params) — the completion count for that goal's single step

Most goals on a given website share the same date range and carry no goal-specific filters, so steps 1 and 2 above are frequently computing the same denominator and issuing structurally identical completion queries that differ only in which single event/pageview they're matching — exactly the shape ClickHouse can answer in one query via GROUP BY.

Proposed fix

Group goals by (a) whether they carry any filters (request-level or goal-level) and (b) their effective start date:

  • Goals with no filters at all in the same date bucket can be counted in a single batched query — one shared getTotalWebsiteUsers call plus one query that matches all their step conditions and returns per-goal completion counts via GROUP BY, instead of N×2 queries.
  • Goals with filters keep the existing one-query-per-goal path unchanged, since the shared event-stream query builder (buildIdentifiedEventStream) only threads filter conditions through the step at array index 0 (correct for its actual funnel-entry-filter use case) — batching filtered goals together would silently apply another goal's filter to the wrong goal.

This bounds the fix to the case where it's provably correct without touching the shared funnel query builder, while still collapsing the common case (dashboards with mostly unfiltered goals) from O(N) queries to O(distinct date ranges).

Will open a PR shortly with the batched implementation, a regression test asserting the query count, and no change in output for existing behavior.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read packages/rpc/src/routers/goals.ts, focusing on goalsRouter.bulkAnalytics, processGoalConversionCount, and getTotalWebsiteUsers. Trace the existing Promise.all and ClickHouse query helpers first; done means unfiltered goals share batched counts by effective start-date bucket, filtered goals retain the existing path, and a regression test confirms query count and unchanged output.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, data, databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.