bottlerocket-os / bottlerocket-os/bottlerocket
I regularly have a problem with nodes going NotReady.
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
**Image I'm using:**
ami-008f64e1aa6e8255a
**What I expected to happen:**
Kubelet to be stable
**What actually happened:**
Kublet stops posting updates.
**How to reproduce the problem:**
I wish I could reproduce at will, but I'll try to describe what happens:
Right before kubelet stops posting update, total cpu will spike up to close to 100% of the available CPU on the node. iowait will also go up, kubelet cpu usage goes up to 17%. There doesn't seem to be any memory pressure, though, memory usage stays stable. On the attached system volumes, read iops also shoot up to basically the limit, and then the node goes unresponsive. SSM also doesn't work. Rebooting the node usually resolves the issue and I can log in and look at logs. What I see in the kubelet logs is a _lot_ of 'context deadline exceeded' errors and not much else. There are logs from containerd that also show 'context deadline exceeded'
Contributor guide
Assessment
This issue has not been assessed yet.