PostHog / PostHog/posthog

fix(signals): the enablement backfill fans the five newest issues into separate reports and PRs within minutes

Open
#90,189 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

self-driving
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Problem

Enabling error tracking as a signal source runs BackfillErrorTrackingWorkflow (products/signals/backend/temporal/backfill_error_tracking.py), which emits the five most recently created issues from the last 30 days as issue_created signals in one go (BACKFILL_ISSUE_LIMIT = 5, BACKFILL_WINDOW_DAYS). Each signal then walks the normal path: grouping, research, and, when the report is immediately actionable and the team's autostart settings allow it, an implementation task and a PR.

On one customer project the backfill fired at 00:23, four separate reports existed by 00:24, two implementation tasks started at 00:30, and two PRs changing the same file for the same reason were open by 00:49. Twenty-six minutes after turning the product on, the customer's first impression was duplicate reports and duplicate PRs. The five backfilled issues were, by construction, whatever happened to be newest, which on a project with local dev servers reporting into production means dev noise.

Fixes

  1. Hold auto-start for backfilled signals. Tag backfill-originated signals (extra={"backfill": true} in emit_backfill_signal_activity) and have maybe_autostart_implementation_task skip auto-start for reports whose signals are all backfill, with a skip_reason the inbox can show. The reports still generate and notify; the customer presses Create PR if they want one.
  2. Let the batch settle before research. Run grouping for the five backfilled signals as one batch (it may already be one batch; confirm) and delay research on them by a short window so the sibling overlap check and any later grouping matches can apply before PRs open.
  3. Prefer issues with production impact. Instead of the five newest issues, pick by occurrence count and distinct users in the window, or at least exclude issues whose events all come from local development hosts once #90178 makes that visible.
  4. Same posture for other backfills (health checks, replay clusters) if they emit on enable.
  5. Tests: backfill-tagged report does not auto-start; a report with one backfill and one live signal behaves as today; the issue selection query prefers impact over recency.

Why

The first hour after enablement is when the product is judged. A damper on auto-start and a smarter pick of seed issues costs nothing in coverage and removes the worst-case first impression.

From this Slack thread; the investigation is written up in the self-driving-debugging skill in the skills store.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with products/signals/backend/temporal/backfill_error_tracking.py and trace emit_backfill_signal_activity into maybe_autostart_implementation_task; confirm whether grouping already batches backfilled signals. Done means backfill-only reports do not auto-start, mixed reports retain current behavior, and selection favors production impact rather than recency, with matching tests added.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.