VictoriaMetrics / VictoriaMetrics/operator
vmalert: vmrule changes could trigger unrelated group restarts during reload in large environments
@AndrewChubatiuk is already working on this.
Since Sep 14, 2026.
- Dominant language
- Go
- Stars
- 589
- Forks
- 229
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 77
Description
Describe the bug
In https://github.com/VictoriaMetrics/operator/commit/4ce88cc4036b20141b4d1370aa62efbfb7c208ce, VMRule groups started being packed into gzip-compressed ConfigMap buckets. But bucket membership is recomputed from scratch on every reconcile, adding or deleting one VMRule can move other untouched groups into a different bucket file.
And since vmalert takes the file path into account when computing the group ID, the moved group is treated as a brand-new group: the old one is stopped and a new one is started, which breaks the existing periodic evaluation and causes the group’s rules to lose their state(also includes their metrics).
The default VM_CONFIG_DATA_BUDGET_BYTES is 524288, so groups of most setups should stay within a single ConfigMap and are unaffected. But for large environments with thousands of rules, multiple configmaps(buckets) can be created, every add or delete can reshuffle unrelated groups and restart them.
Previously, this case should be more rarer, as rule files were packed by sorted filename with sequential fill. Files sorted before the changed file could never move, only a few files after it could switch files if there was new free room for them.
To Reproduce
Set VM_CONFIG_DATA_BUDGET_BYTES to a small value and create, for example, 12 VMRule resources so they are split into 4 ConfigMaps (buckets). Each ConfigMap will contain 3 VMRule resources, and deleting any of them will change the distribution from [3 3 3 3] to [4 4 3], even deleting the last one, which would not have affected the distribution before.
Expected behavior
Rule groups are more resistant to random vmrule change.
Version
Since v0.72.0.
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.