VictoriaMetrics / VictoriaMetrics/operator
Multiple statefulsets for vmstorage nodes on vmcluster
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 589
- Forks
- 229
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 77
Description
Currently operator creates one statefulset for all vmstorage nodes and scales it accordingly. This approach is good until something bad happens to one of storage nodes. For example:
It is not possible to turn off one of vmstorage nodes (for restore purpose)
We cannot keep the corrupted vmstorage for diagnosis while adding another one to keep cluster running.
We cannot change resource limits seperately. This is specially needed during incidents for temporarily adjustments. Moreover, we have seen that some storages behave differently in our cluster, this can happen because of different computing infrastructure or maybe a flaw in load balancing in shard ingestion or search loads.
We propose that the operator deploy each storage as one separate sts. In this way maintenances would be simpler, and also there would be a way to set different resoruce configuration on vmcluster CRD object.
You might have concerns about the simplicity. We can think of a more felxible representation of vmcluster. If vmoperator can support different nodepools in defining vmcluster it can keep it simple to new users as well as allowing people for more advanced designs.
For example:
vmstorage:
nodepools:
- name: pool1
replicaCount: 2
resources: {}
tolerations: {}
nodeSelector {}
- name: pool2
replicaCount: 1
resources: {}
tolerations: {}
nodeSelector {}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the operator entry points that create the vmstorage StatefulSet and define the vmcluster CRD. Review how replica counts, resources, tolerations, and node selectors are currently represented, then determine the nodepool semantics needed for separate storage StatefulSets. Done means the proposal is specified well enough to support independent maintenance and resource configuration for each storage group.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100