VictoriaMetrics / VictoriaMetrics/operator

BUG: vmalert config-reloader fails to start when spec.configMaps is set (operator v0.74.1)

Open
#2,583 3 comments 0 reactions 1 assignee View on GitHub

@AndrewChubatiuk is already working on this.

Since Sep 16, 2026.

bug
Dominant language
Go
Stars
589
Forks
229
Avg merge
1d 9h
Merged PRs (30d)
77

Description

Describe the bug

The operator builds a config-reloader sidecar spec that its own reloader binary rejects.

When a VMAlert sets spec.configMaps, the sidecar crashes on start:

--target-dir count (1) must match watched directory count (2) from --watched-dir/--rules-dir

The pod never becomes ready. The rollout is stuck.

To Reproduce

  1. Run operator v0.74.1 with config-reloader-v0.74.1.
  2. Create a ConfigMap my-templates with a .tmpl file.
  3. Create a VMAlert with:
spec:
  configMaps:
    - my-templates
  extraArgs:
    rule.templates: /etc/vm/configs/**/*.tmpl
  1. The config-reloader container fails. Exit code 2.

Generated args:

--target-dir=/etc/vmalert/rules-out/rules-src-0
--watched-dir=/etc/vm/configs/my-templates
--watched-dir=/etc/vmalert/rules-src-0

Two watched dirs, one target dir.

Root cause

Two separate problems, both in v0.74.1.

1. Unpaired --watched-dir

internal/controller/operator/factory/build/container.go:418-422 adds a bare --watched-dir for every entry in spec.configMaps. No --target-dir goes with it.

AddWatchTargetDir at container.go:525-536 does it correctly. Its own comment says it appends the pair "atomically".

cmd/config-reloader/main.go:100 then calls validateTargetDirCount, which fails at main.go:400-403.

So the operator and its reloader disagree inside a single release.

2. sort.Strings destroys the pairing

internal/controller/operator/factory/vmalert/vmalert.go:355 calls sort.Strings(crc.Args).

The flags are matched by position. Sorting puts /etc/vm/configs/... before /etc/vmalert/rules-src-0. Even with a matching count, the templates would be written into the rules output directory.

The strict check added in reloader v0.73.0 is what stops this. On reloader v0.72.0 the flag exists but there is no validation, so the corruption would be silent.

Why the init container is fine

container.go:416 takes the --only-init-config branch. That branch does not emit the crMounts watched dirs at all. So only the sidecar is affected.

Related

#1370 asked for spec.configMaps to be watched by the reloader. The bare --watched-dir comes from that. The validation added later in the reloader makes the two incompatible.

Workaround

Move the ConfigMap out of spec.configMaps into spec.volumes plus spec.volumeMounts at the same path:

spec:
  configMaps: []
  volumes:
    - name: my-templates
      configMap:
        name: my-templates
  volumeMounts:
    - name: my-templates
      mountPath: /etc/vm/configs/my-templates
      readOnly: true

The vmalert container still sees the files. The reloader gets no extra watched dir. Templates are static, so no reload is needed.

Suggested fix

Use AddWatchTargetDir for crMounts too, or drop the bare --watched-dir loop. Also replace the positional pairing with something order independent, so sort.Strings cannot break it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.