[Notifications] Five-step reload pipeline (lock, render, validate, apply, verify)
- Dominant language
- Go
- Stars
- 55
- Forks
- 16
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 87
Description
The pipeline that takes saves from the API and atomically applies them to the live vmalert and Alertmanager configs, with rollback on any failure.
Checklist:
- [ ] `notifications.ReloadCoordinator` owns a `sync.Mutex` plus the monotonic generation counter from `notification_config_state`. All Create/Update/Delete RPCs go through the coordinator.
- [ ] Renderer produces YAML for the union of (rules across all orgs) and (alertmanager receivers + routes) into temp files alongside the live config.
- [ ] External validation step: `amtool check-config` and `vmalert -dryRun` (binaries packaged in the `fleet-api` image, or invoked via a small sidecar — pick one and document). Validator stderr surfaces verbatim through the API response on failure.
- [ ] On both validators passing: `os.Rename(temp, live)` (same filesystem, atomic) and POST to `vmalert /-/reload` and `alertmanager /-/reload`.
- [ ] Verify success by polling `vmalert_alerting_rules_last_evaluation_timestamp_seconds` and `alertmanager_config_last_reload_successful` for the new generation. 10-second timeout.
- [ ] On any failure, restore from `.previous`, reload again, mark the offending row `validation_status=invalid`, return a clear error.
- [ ] Coordinator exposes `LastGeneration()` and `LastReloadStatus()` for the UI.
- [ ] E2E fault-injection test: corrupt the rendered config mid-pipeline, assert the previous live config is restored.
Contributor guide
Assessment
This issue has not been assessed yet.