openshift / openshift/machine-config-operator

allow setting label to nodes about to be upgraded/restarted

Open
#3,204 8 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

lifecycle/frozen
Dominant language
Go
Stars
269
Forks
529
Avg merge
3d 18h
Merged PRs (30d)
83

Description

Description

Because there is no agreed-upon way to signal operators that a node is drained, there are multiple ways that operators handle it.
Rook detects node drain by observing pods on the node. This works fine but feels a bit fragile.
The problem is that some operators (e.g. the Zalando PostgreSQL Operator) "detect" drains by watching node's labels. Whenever a label is not set anymore (e.g. "node-ready=true") it will (try to) failover to another DB pod on another node.

This is a feature request to update node's labels when a reboot is about to happen.

Steps to reproduce the issue:

  1. update some machineconfig,
  2. observe machine-config-daemon trying to drain a node,
  3. failing to drain the node because there is a pdb on a pod on that node,

meanwhile
4. some operator not knowing that the machine is about to be rebooted and not updating the pdb (directly or indirectly.)

  1. the node not getting drained.

Describe the results you expected:

  1. update some machineconfig,
  2. machine-config-daemon updating label machineconfiguration.openshift.io/pending-restart=false to =true,
    3a. an operator removes active workload from the node, removing/updating pdbs that affect the node,
    3b. machine-config-daemon drains the node,
  3. node reboots successful,
  4. machine-config-daemon sets label machineconfiguration.openshift.io/pending-restart=false.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the machine-config-daemon workflow described in the issue, especially the node drain and reboot sequence triggered by a MachineConfig update. Trace where drain attempts begin and where reboot completion is recorded; done means the pending-restart label changes before draining and returns to false after a successful reboot, with behavior verified around PDB-blocked drains.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.