bottlerocket-os / bottlerocket-os/bottlerocket

I regularly have a problem with nodes going NotReady.

Open
#3,596 5 comments 0 reactions 0 assignees View on GitHub
status/needs-info type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

**Image I'm using:**
ami-008f64e1aa6e8255a

**What I expected to happen:**

Kubelet to be stable

**What actually happened:**

Kublet stops posting updates.

**How to reproduce the problem:**

I wish I could reproduce at will, but I'll try to describe what happens:

Right before kubelet stops posting update, total cpu will spike up to close to 100% of the available CPU on the node. iowait will also go up, kubelet cpu usage goes up to 17%. There doesn't seem to be any memory pressure, though, memory usage stays stable. On the attached system volumes, read iops also shoot up to basically the limit, and then the node goes unresponsive. SSM also doesn't work. Rebooting the node usually resolves the issue and I can log in and look at logs. What I see in the kubelet logs is a _lot_ of 'context deadline exceeded' errors and not much else. There are logs from containerd that also show 'context deadline exceeded'

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.