VictoriaMetrics / VictoriaMetrics/operator

Bug: VMRule `spec.groups[].tenant` should accept an integer, not only a string

Open Beginner friendly
#2,608 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
589
Forks
229
Avg merge
1d 9h
Merged PRs (30d)
77

Description

What happened

A VMRule in the cluster carried a tenant id written as a bare YAML number:

spec:
  groups:
    - name: some-group
      tenant: 0          # number, not "0"
      rules: [...]

VMRuleGroup.Tenant is typed string (api/operator/v1beta1/vmrule_types.go:70), so the
typed LIST that backs the VMRule informer fails to decode:

{"level":"error","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch",
 "reflector":"pkg/mod/k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285",
 "type":"*v1beta1.VMRule",
 "error":"failed to list *v1beta1.VMRule: json: cannot unmarshal number into Go struct field RuleGroup.items.spec.groups.tenant of type string"}

Go's decoder rejects the whole list response, not the single offending item, so the
VMRule informer never syncs.

Impact

Because WaitForCacheSync blocks manager startup, that one object stops the operator process
from ever becoming ready. On an operator with several CRs in flight, the observed failure is a
crash loop with a cascade of secondary errors that do not name the real cause (names and
namespaces redacted):

{"logger":"manager","msg":"error received after stop sequence was engaged",
 "error":"failed to wait for vlagent caches to sync kind source: *v1.ServiceAccount: timed out waiting for cache to be synced for Kind *v1.ServiceAccount"}

{"logger":"manager","msg":"Reconciler error","controller":"vmagent",
 "error":"failed to update object status: \"cannot update resource status with patch: client rate limiter Wait returned an error: context canceled\", origin err: cannot create vmagent role and binding for it, err: cannot perform RBAC migration: cannot get object: Timeout: failed waiting for *v1.ClusterRole Informer to sync"}

{"logger":"manager","msg":"Reconciler error","controller":"vmauth",
 "error":"... origin err: cannot create or update vmauth deploy: cannot create or update ingress for vmauth: Timeout: failed waiting for *v1.Ingress Informer to sync"}

{"logger":"manager","msg":"Reconciler error","controller":"vmalert",
 "error":"... origin err: cannot list objects at cluster scope: Timeout: failed waiting for *v1beta1.VMRule Informer to sync"}

{"logger":"setup","msg":"cannot setup manager",
 "error":"cannot start controller manager: failed to wait for vmpodscrape caches to sync kind source: *v1beta1.VMPodScrape: timed out waiting for cache to be synced for Kind *v1beta1.VMPodScrape"}

cannot setup manager reaches os.Exit(1) (internal/manager/manager.go:360 ->
cmd/main.go:44), so the operator CrashLoopBackOffs. A single VMRule in one namespace stops
reconciliation of every managed object cluster-wide, and none of the emitted errors point at the
VMRule that caused it.

Workaround is to quote the value (tenant: "0").

Proposal

Accept both forms for tenant, the way the operator already does for other scalars that users
naturally write unquoted.

Environment

  • operator v0.63.0, chart victoria-metrics-operator, 2 replicas, leader election enabled
  • watching all namespaces
  • client-go v0.33.2 per the reflector log line

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with VMRuleGroup.Tenant in api/operator/v1beta1/vmrule_types.go:70 and trace how VMRule objects are decoded into the informer. Check the existing VMRule API definitions and related tests, then verify that both quoted strings and bare integer tenant values decode successfully without preventing the VMRule informer from syncing.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, backend, devops
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.