VictoriaMetrics / VictoriaMetrics/operator

Multiple statefulsets for vmstorage nodes on vmcluster

Open
#741 8 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Go
Stars
589
Forks
229
Avg merge
1d 9h
Merged PRs (30d)
77

Description

Currently operator creates one statefulset for all vmstorage nodes and scales it accordingly. This approach is good until something bad happens to one of storage nodes. For example:
It is not possible to turn off one of vmstorage nodes (for restore purpose)
We cannot keep the corrupted vmstorage for diagnosis while adding another one to keep cluster running.
We cannot change resource limits seperately. This is specially needed during incidents for temporarily adjustments. Moreover, we have seen that some storages behave differently in our cluster, this can happen because of different computing infrastructure or maybe a flaw in load balancing in shard ingestion or search loads.
We propose that the operator deploy each storage as one separate sts. In this way maintenances would be simpler, and also there would be a way to set different resoruce configuration on vmcluster CRD object.
You might have concerns about the simplicity. We can think of a more felxible representation of vmcluster. If vmoperator can support different nodepools in defining vmcluster it can keep it simple to new users as well as allowing people for more advanced designs.
For example:

vmstorage:
  nodepools:
    - name: pool1
       replicaCount: 2
       resources: {}
       tolerations: {}
       nodeSelector {}
    - name: pool2
       replicaCount: 1
       resources: {}
       tolerations: {}
       nodeSelector {}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the operator entry points that create the vmstorage StatefulSet and define the vmcluster CRD. Review how replica counts, resources, tolerations, and node selectors are currently represented, then determine the nodepool semantics needed for separate storage StatefulSets. Done means the proposal is specified well enough to support independent maintenance and resource configuration for each storage group.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.