bottlerocket-os / bottlerocket-os/bottlerocket-core-kit

Pluto service takes 5 minutes to commit settings

Open
#267 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
34
Forks
77
Avg merge
2d 23h
Merged PRs (30d)
23

Description

Making this bug report because we observe slow init time for bottlerocket EKS nodes. Nodes get stuck in ~5 minutes before starting kubelet and joining the cluster. So far we pinpointed the source to slow pluto.service commit stage (which seems to come from this repo, correct?).

The clusters are using the latest EKS-optimized Bottlerocket image. It reproduces consistently on every new node but not for every cluster.

The question is how to investigate and fix the cause of this? We are not sure if this is a package issue or configuration issue in the clusters. The clusters have IDMS enabled. Not sure what else is required for this process.

**Package I'm using:**
`pluto.service`

**What I expected to happen:**
Startup to take 1-2 minutes and not 5+ minutes.

**What actually happened:**
Looking at systemd logs, `pluto.service` took 5 minutes to complete. We extracted logs from it and we observe the `Committing settings` step taking 5 minutes.

Logs from pluto:
```
bash-5.0# journalctl -u pluto.service
Nov 14 07:42:00 localhost systemd[1]: Starting Generate additional settings for Kubernetes...
Nov 14 07:42:00 localhost settings-committer[1832]: 07:42:00 [INFO] Checking pending settings.
Nov 14 07:42:00 localhost settings-committer[1832]: 07:42:00 [INFO] Committing settings.
Nov 14 07:47:01 localhost systemd[1]: Finished Generate additional settings for Kubernetes.
```

**How to reproduce the problem:**
Unclear, we only see this issue in some customer clusters but not on a fresh cluster.

** Extra information **
bash-5.0# apiclient get os
{
"os": {
"arch": "x86_64",
"build_id": "360b7a38",
"pretty_name": "Bottlerocket OS 1.26.2 (aws-k8s-1.30)",
"variant_id": "aws-k8s-1.30",
"version_id": "1.26.2"
}
}

Contributor guide

Open the contributing guide

Research direction

The report identifies pluto.service and its settings-committer "Committing settings" stage; begin with the systemd journal shown and compare affected versus fresh EKS clusters, including IDMS configuration. Done means the cause is isolated and startup no longer spends five minutes committing settings, with evidence from the service timing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.