block / block/proto-fleet

[Notifications] Five-step reload pipeline (lock, render, validate, apply, verify)

Open
#179 0 comments 0 reactions 1 assignee Claimed by @illegalprime View on GitHub
Dominant language
Go
Stars
55
Forks
16
Avg merge
1d 22h
Merged PRs (30d)
87

Description

The pipeline that takes saves from the API and atomically applies them to the live vmalert and Alertmanager configs, with rollback on any failure.

Checklist:

- [ ] `notifications.ReloadCoordinator` owns a `sync.Mutex` plus the monotonic generation counter from `notification_config_state`. All Create/Update/Delete RPCs go through the coordinator.
- [ ] Renderer produces YAML for the union of (rules across all orgs) and (alertmanager receivers + routes) into temp files alongside the live config.
- [ ] External validation step: `amtool check-config` and `vmalert -dryRun` (binaries packaged in the `fleet-api` image, or invoked via a small sidecar — pick one and document). Validator stderr surfaces verbatim through the API response on failure.
- [ ] On both validators passing: `os.Rename(temp, live)` (same filesystem, atomic) and POST to `vmalert /-/reload` and `alertmanager /-/reload`.
- [ ] Verify success by polling `vmalert_alerting_rules_last_evaluation_timestamp_seconds` and `alertmanager_config_last_reload_successful` for the new generation. 10-second timeout.
- [ ] On any failure, restore from `.previous`, reload again, mark the offending row `validation_status=invalid`, return a clear error.
- [ ] Coordinator exposes `LastGeneration()` and `LastReloadStatus()` for the UI.
- [ ] E2E fault-injection test: corrupt the rendered config mid-pipeline, assert the previous live config is restored.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.