getsentry / getsentry/self-hosted

Pre-existing Issue Alert Rules and Metric Alert Rules silently stop firing after upgrading to 26.7.0+ — never migrated to the Workflow Engine

Open
#4,452 3 comments 0 reactions 0 assignees View on GitHub
Waiting for: Product Owner
Dominant language
Shell
Stars
9.6k
Forks
2k
Avg merge
1d 5h
Merged PRs (30d)
15

Description

### Self-Hosted Version

26.7.1

### CPU Architecture

x86_64

### Docker Version

N/A

### Docker Compose Version

N/A

### Machine Specification

- [x] My system meets the minimum system requirements of Sentry

### Installation Type

Upgrade from 26.5.0 to 26.7.1

### Steps to Reproduce

1. Run a self-hosted instance predating the Workflow Engine, with existing Issue Alert Rules and/or Metric Alert Rules on various projects.
2. Upgrade to 26.7.0+, which enables the Workflow Engine (getsentry/self-hosted#4364).
3. Don't re-create or re-edit any pre-existing alert rules after the upgrade.
4. Check the Alerts UI for a project with an untouched pre-existing rule, and check whether that project has a default Error Monitor detector.

### Expected Result

Pre-existing Issue Alert Rules and Metric Alert Rules keep firing after the upgrade — either auto-migrated to the new Workflow/Detector model, or still working via the old model as a fallback.

### Actual Result

`Rule` and `AlertRule` objects are never migrated to the new `Workflow`/`Detector` model unless a human creates or edits that specific rule again after the upgrade. No automatic retroactive migration exists:

- The project-creation signal (`workflow_engine/receivers/project_detectors.py`) only fires for *new* projects.
- The one backfill migration touching existing data (`workflow_engine/migrations/0094_backfill_issue_stream_detector_workflows.py`) only operates on projects that *already have* an error detector — it can't create a first one.
- That migration and the equivalent for metric alerts are marked `is_post_deployment = True`, meant to be triggered manually post-deploy per Sentry's own convention. Self-hosted's install/upgrade tooling has no such trigger anywhere.

Impact on our instance (~430 active projects, ~1,300 active alert rules):

- **52 of 1,169** active Issue Alert Rules, across **24 projects**, silently unmigrated and dead.
- **5 of 143** active Metric Alert Rules likewise unmigrated and dead.
- **19 of 432** active projects had zero default Error Monitor — blocking even *new* alert creation, since the UI only attaches alerts to an existing monitor and Error Monitors can't be created manually.

All silent — no error, no log, just an empty alert list.

**Workaround:** manually invoked `IssueAlertMigrator`/`dual_write_alert_rule` per affected rule via `sentry django shell`, gated on a "not already migrated" check (neither function is idempotent on its own). Worked cleanly for all 57 rules.

**Suggested fix:** document this in the upgrade guide, or ship an idempotent management command that backfills pre-existing `Rule`/`AlertRule` objects post-upgrade. Happy to help shape/test a fix.

Related (different symptoms, same 26.7.0 rollout): #4385, #4105, #4110, #4341, #4044

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading workflow_engine/receivers/project_detectors.py and migration 0094_backfill_issue_stream_detector_workflows.py, then inspect the equivalent metric-alert migration and the IssueAlertMigrator/dual_write_alert_rule entry points via sentry django shell. Done means pre-existing Issue Alert Rules and Metric Alert Rules are migrated after upgrade, including projects without an existing default Error Monitor, and continue firing without manual per-rule intervention.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
backend, databases, release
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.