VictoriaMetrics / VictoriaMetrics/operator

vmalert: vmrule changes could trigger unrelated group restarts during reload in large environments

Open
#2,610 0 comments 0 reactions 1 assignee View on GitHub

@AndrewChubatiuk is already working on this.

Since Sep 14, 2026.

Dominant language
Go
Stars
589
Forks
229
Avg merge
1d 9h
Merged PRs (30d)
77

Description

Describe the bug

In https://github.com/VictoriaMetrics/operator/commit/4ce88cc4036b20141b4d1370aa62efbfb7c208ce, VMRule groups started being packed into gzip-compressed ConfigMap buckets. But bucket membership is recomputed from scratch on every reconcile, adding or deleting one VMRule can move other untouched groups into a different bucket file.
And since vmalert takes the file path into account when computing the group ID, the moved group is treated as a brand-new group: the old one is stopped and a new one is started, which breaks the existing periodic evaluation and causes the group’s rules to lose their state(also includes their metrics).

The default VM_CONFIG_DATA_BUDGET_BYTES is 524288, so groups of most setups should stay within a single ConfigMap and are unaffected. But for large environments with thousands of rules, multiple configmaps(buckets) can be created, every add or delete can reshuffle unrelated groups and restart them.

Previously, this case should be more rarer, as rule files were packed by sorted filename with sequential fill. Files sorted before the changed file could never move, only a few files after it could switch files if there was new free room for them.

To Reproduce

Set VM_CONFIG_DATA_BUDGET_BYTES to a small value and create, for example, 12 VMRule resources so they are split into 4 ConfigMaps (buckets). Each ConfigMap will contain 3 VMRule resources, and deleting any of them will change the distribution from [3 3 3 3] to [4 4 3], even deleting the last one, which would not have affected the distribution before.

Expected behavior

Rule groups are more resistant to random vmrule change.

Version

Since v0.72.0.

Additional context

Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.