bottlerocket-os / bottlerocket-os/bottlerocket-update-operator

Nodes never uncordoned

Open
#691 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
220
Forks
50
PR merge metrics
No merged PRs in 30d

Description

**Image I'm using:**
v1.4.0

**Issue:**
We're seeing our nodes get stuck with the taint `node.kubernetes.io/unschedulable:NoSchedule` after an update. It doesn't look like this happens to every node when it updates, only some of them. When it does hit this issue - the node successfully performs the update, reboots, but is never uncordoned when it comes back up.

The last thing in the controller logs is the event `RebootedIntoUpdate` and I see the node reports the new version of bottlerocket
```
status: Some(BottlerocketShadowStatus { current_version: \"1.24.0\", target_version: \"1.25.0\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None })
```

Last event I can see in the agent logs is `Bottlerocket node is terminated by reboot signal`

** Helm Values **
```
scheduler_cron_expression: "0 0 8 * * Sun"
logging:
formatter: json
prometheus:
controller:
serviceMonitor:
enabled: true
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.