getsentry / getsentry/self-hosted
Pre-existing Issue Alert Rules and Metric Alert Rules silently stop firing after upgrading to 26.7.0+ — never migrated to the Workflow Engine
- Dominant language
- Shell
- Stars
- 9.6k
- Forks
- 2k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 15
Description
### Self-Hosted Version
26.7.1
### CPU Architecture
x86_64
### Docker Version
N/A
### Docker Compose Version
N/A
### Machine Specification
- [x] My system meets the minimum system requirements of Sentry
### Installation Type
Upgrade from 26.5.0 to 26.7.1
### Steps to Reproduce
1. Run a self-hosted instance predating the Workflow Engine, with existing Issue Alert Rules and/or Metric Alert Rules on various projects.
2. Upgrade to 26.7.0+, which enables the Workflow Engine (getsentry/self-hosted#4364).
3. Don't re-create or re-edit any pre-existing alert rules after the upgrade.
4. Check the Alerts UI for a project with an untouched pre-existing rule, and check whether that project has a default Error Monitor detector.
### Expected Result
Pre-existing Issue Alert Rules and Metric Alert Rules keep firing after the upgrade — either auto-migrated to the new Workflow/Detector model, or still working via the old model as a fallback.
### Actual Result
`Rule` and `AlertRule` objects are never migrated to the new `Workflow`/`Detector` model unless a human creates or edits that specific rule again after the upgrade. No automatic retroactive migration exists:
- The project-creation signal (`workflow_engine/receivers/project_detectors.py`) only fires for *new* projects.
- The one backfill migration touching existing data (`workflow_engine/migrations/0094_backfill_issue_stream_detector_workflows.py`) only operates on projects that *already have* an error detector — it can't create a first one.
- That migration and the equivalent for metric alerts are marked `is_post_deployment = True`, meant to be triggered manually post-deploy per Sentry's own convention. Self-hosted's install/upgrade tooling has no such trigger anywhere.
Impact on our instance (~430 active projects, ~1,300 active alert rules):
- **52 of 1,169** active Issue Alert Rules, across **24 projects**, silently unmigrated and dead.
- **5 of 143** active Metric Alert Rules likewise unmigrated and dead.
- **19 of 432** active projects had zero default Error Monitor — blocking even *new* alert creation, since the UI only attaches alerts to an existing monitor and Error Monitors can't be created manually.
All silent — no error, no log, just an empty alert list.
**Workaround:** manually invoked `IssueAlertMigrator`/`dual_write_alert_rule` per affected rule via `sentry django shell`, gated on a "not already migrated" check (neither function is idempotent on its own). Worked cleanly for all 57 rules.
**Suggested fix:** document this in the upgrade guide, or ship an idempotent management command that backfills pre-existing `Rule`/`AlertRule` objects post-upgrade. Happy to help shape/test a fix.
Related (different symptoms, same 26.7.0 rollout): #4385, #4105, #4110, #4341, #4044
### Event ID
_No response_
Contributor guide
Research direction
Start by reading workflow_engine/receivers/project_detectors.py and migration 0094_backfill_issue_stream_detector_workflows.py, then inspect the equivalent metric-alert migration and the IssueAlertMigrator/dual_write_alert_rule entry points via sentry django shell. Done means pre-existing Issue Alert Rules and Metric Alert Rules are migrated after upgrade, including projects without an existing default Error Monitor, and continue firing without manual per-rule intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- django, python
- Domain
- backend, databases, release
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100