bottlerocket-os / bottlerocket-os/bottlerocket-core-kit
Pluto service takes 5 minutes to commit settings
- Dominant language
- Rust
- Stars
- 34
- Forks
- 77
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 23
Description
Making this bug report because we observe slow init time for bottlerocket EKS nodes. Nodes get stuck in ~5 minutes before starting kubelet and joining the cluster. So far we pinpointed the source to slow pluto.service commit stage (which seems to come from this repo, correct?).
The clusters are using the latest EKS-optimized Bottlerocket image. It reproduces consistently on every new node but not for every cluster.
The question is how to investigate and fix the cause of this? We are not sure if this is a package issue or configuration issue in the clusters. The clusters have IDMS enabled. Not sure what else is required for this process.
**Package I'm using:**
`pluto.service`
**What I expected to happen:**
Startup to take 1-2 minutes and not 5+ minutes.
**What actually happened:**
Looking at systemd logs, `pluto.service` took 5 minutes to complete. We extracted logs from it and we observe the `Committing settings` step taking 5 minutes.
Logs from pluto:
```
bash-5.0# journalctl -u pluto.service
Nov 14 07:42:00 localhost systemd[1]: Starting Generate additional settings for Kubernetes...
Nov 14 07:42:00 localhost settings-committer[1832]: 07:42:00 [INFO] Checking pending settings.
Nov 14 07:42:00 localhost settings-committer[1832]: 07:42:00 [INFO] Committing settings.
Nov 14 07:47:01 localhost systemd[1]: Finished Generate additional settings for Kubernetes.
```
**How to reproduce the problem:**
Unclear, we only see this issue in some customer clusters but not on a fresh cluster.
** Extra information **
bash-5.0# apiclient get os
{
"os": {
"arch": "x86_64",
"build_id": "360b7a38",
"pretty_name": "Bottlerocket OS 1.26.2 (aws-k8s-1.30)",
"variant_id": "aws-k8s-1.30",
"version_id": "1.26.2"
}
}
Contributor guide
Research direction
The report identifies pluto.service and its settings-committer "Committing settings" stage; begin with the systemd journal shown and compare affected versus fresh EKS clusters, including IDMS configuration. Done means the cause is isolated and startup no longer spends five minutes committing settings, with evidence from the service timing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100