VictoriaMetrics / VictoriaMetrics/operator
Bug: VMRule `spec.groups[].tenant` should accept an integer, not only a string
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 589
- Forks
- 229
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 77
Description
What happened
A VMRule in the cluster carried a tenant id written as a bare YAML number:
spec:
groups:
- name: some-group
tenant: 0 # number, not "0"
rules: [...]
VMRuleGroup.Tenant is typed string (api/operator/v1beta1/vmrule_types.go:70), so the
typed LIST that backs the VMRule informer fails to decode:
{"level":"error","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch",
"reflector":"pkg/mod/k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285",
"type":"*v1beta1.VMRule",
"error":"failed to list *v1beta1.VMRule: json: cannot unmarshal number into Go struct field RuleGroup.items.spec.groups.tenant of type string"}
Go's decoder rejects the whole list response, not the single offending item, so the
VMRule informer never syncs.
Impact
Because WaitForCacheSync blocks manager startup, that one object stops the operator process
from ever becoming ready. On an operator with several CRs in flight, the observed failure is a
crash loop with a cascade of secondary errors that do not name the real cause (names and
namespaces redacted):
{"logger":"manager","msg":"error received after stop sequence was engaged",
"error":"failed to wait for vlagent caches to sync kind source: *v1.ServiceAccount: timed out waiting for cache to be synced for Kind *v1.ServiceAccount"}
{"logger":"manager","msg":"Reconciler error","controller":"vmagent",
"error":"failed to update object status: \"cannot update resource status with patch: client rate limiter Wait returned an error: context canceled\", origin err: cannot create vmagent role and binding for it, err: cannot perform RBAC migration: cannot get object: Timeout: failed waiting for *v1.ClusterRole Informer to sync"}
{"logger":"manager","msg":"Reconciler error","controller":"vmauth",
"error":"... origin err: cannot create or update vmauth deploy: cannot create or update ingress for vmauth: Timeout: failed waiting for *v1.Ingress Informer to sync"}
{"logger":"manager","msg":"Reconciler error","controller":"vmalert",
"error":"... origin err: cannot list objects at cluster scope: Timeout: failed waiting for *v1beta1.VMRule Informer to sync"}
{"logger":"setup","msg":"cannot setup manager",
"error":"cannot start controller manager: failed to wait for vmpodscrape caches to sync kind source: *v1beta1.VMPodScrape: timed out waiting for cache to be synced for Kind *v1beta1.VMPodScrape"}
cannot setup manager reaches os.Exit(1) (internal/manager/manager.go:360 ->
cmd/main.go:44), so the operator CrashLoopBackOffs. A single VMRule in one namespace stops
reconciliation of every managed object cluster-wide, and none of the emitted errors point at the
VMRule that caused it.
Workaround is to quote the value (tenant: "0").
Proposal
Accept both forms for tenant, the way the operator already does for other scalars that users
naturally write unquoted.
Environment
- operator
v0.63.0, chartvictoria-metrics-operator, 2 replicas, leader election enabled - watching all namespaces
client-go v0.33.2per the reflector log line
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with VMRuleGroup.Tenant in api/operator/v1beta1/vmrule_types.go:70 and trace how VMRule objects are decoded into the informer. Check the existing VMRule API definitions and related tests, then verify that both quoted strings and bare integer tenant values decode successfully without preventing the VMRule informer from syncing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- api, backend, devops
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100