VictoriaMetrics / VictoriaMetrics/operator

config-reloader ContainerPort ignores configReloaderExtraArgs http.listenAddr (hostNetwork port conflict)

Open
#2,584 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
589
Forks
229
Avg merge
1d 9h
Merged PRs (30d)
77

Description

Summary

The operator always sets containerPort to the constant 8435 (ConfigReloaderDefaultPort in internal/controller/operator/factory/build/container.go) when building the config-reloader sidecar. Changing the process listen address with spec.configReloaderExtraArgs["http.listenAddr"] updates container args only. Neither containerPort nor the generated HTTP liveness/readiness probes respect the overridden port — both stay locked to 8435, so the kube-scheduler still sees a host port conflict even when the process listens elsewhere.

On nodes with hostNetwork: true, that blocks scheduling a second DaemonSet that also needs a config-reloader: Kubernetes reports that the node did not have free ports for the requested pod ports (didn't have free ports…), even when one process listens on 8435 and another on 8436 via http.listenAddr.

CommonConfigReloaderParams is shared across VMAgent, VMAuth, VMSingle, VMAlertmanager, and VMAlert. The bug lives in the common builder build.ConfigReloaderContainer and affects any component that uses configReloaderExtraArgs to change the listen address.

Environment

  • Operator: v0.74.0 (also reproduced in source of v0.74.1, v0.75.0-rc1, and current master as of 2026-09-04)
  • Resources: two VMAgent DaemonSets with hostNetwork: true on the same node
  • Sidecar: default VictoriaMetrics config-reloader

Steps to reproduce

  1. Deploy two VMAgent CRs as DaemonSets with hostNetwork: true (and otherwise valid scrape config).
  2. On the second agent, override the reloader listen port, for example:
spec:
  hostNetwork: true
  configReloaderExtraArgs:
    http.listenAddr: "127.0.0.1:8436"
  1. Observe PodSpec of the second DaemonSet: config-reloader still has containerPort: 8435 and probes targeting 8435, while args include --http.listenAddr=127.0.0.1:8436.
  2. Scheduler fails to place the second pod on a node that already runs the first agent’s reloader on host port 8435.

Expected

ContainerPort and HTTP probe ports for config-reloader should follow the port from configReloaderExtraArgs["http.listenAddr"] when set (with fallback to 8435).

Actual

Process listens on the overridden address (127.0.0.1:8436). PodSpec still declares containerPort: 8435 and probes 8435. Scheduler rejects the pod on a node that already exposes host port 8435 for another agent’s reloader.

Proposed fix

  1. Parse the port from ConfigReloaderExtraArgs["http.listenAddr"] (support :PORT, IP:PORT, [IPv6]:PORT).
  2. Use that port for ContainerPort and liveness/readiness probes in addPortProbesToConfigReloaderContainer (default 8435 if unset/invalid).
  3. Align scrape endpoint builders that hardcode 8435 if they scrape the sidecar by numeric port.
  4. Update configReloaderMetricsURL in internal/controller/operator/factory/reconcile/config_reload.go, which also hardcodes build.ConfigReloaderDefaultPort (8435). Controllers that call WaitForConfigReloadHash (e.g. VMAuth) poll /metrics on that URL; when the sidecar listen port is overridden, they keep polling 8435 and eventually time out. configReloaderMetricsURL should resolve the configured port the same way as ContainerPort and probes.
  5. Add unit tests in container_test.go for default, overridden, IPv6, and invalid values.

Related

  • VictoriaMetrics/operator#1581 — IPv6 / enableTCP6 / http.listenAddr interaction (different symptom; related listen-address handling).
  • VictoriaMetrics/operator#1308 — hardcoded config-reloader probes with proxy-protocol (probe port behavior; not hostNetwork / ContainerPort scheduling).

Related code

  • internal/controller/operator/factory/build/container.go: ConfigReloaderDefaultPort, ConfigReloaderContainer, addPortProbesToConfigReloaderContainer

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in internal/controller/operator/factory/build/container.go by reading ConfigReloaderContainer, addPortProbesToConfigReloaderContainer, and ConfigReloaderDefaultPort, then inspect internal/controller/operator/factory/reconcile/config_reload.go for configReloaderMetricsURL. Add the requested cases to container_test.go and verify that container ports, probes, and metrics polling use the configured port with the documented fallback and address forms.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.