bottlerocket-os / bottlerocket-os/bottlerocket

Misconfigured User-data would prevent Bottlerocket to join cluster.

Open
#4,779 2 comments 0 reactions 0 assignees View on GitHub
status/needs-triage type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

**Image I'm using:**

* `bottlerocket-aws-k8s-1.35-x86_64-v1.55.0-d93bb1b1`

**What I expected to happen:**

Misconfigured User-data would prevent Bottlerocket to join cluster.

**What actually happened:**

1. Launch Bottlerocket with the following User-data

```toml
settings.kubernetes.cluster-name = 'eks-cluster-debug'
settings.kubernetes.api-server = 'https://EXAMPLE.us-east-1.eks.amazonaws.com'
settings.kubernetes.cluster-certificate = 'EXAMPLE'
settings.kubernetes.cluster-dns-ip = 'EXAMPLE'
settings.kubernetes.max-pods = 110
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup' = 'bottlerocket-1'
# ... (OMITTED)

settings.aws.region = "us-east-2" # <------------ Key to the issue.
```

2. A misconfigured [settings.aws.region](https://bottlerocket.dev/en/os/1.54.x/api/settings/aws/#region) in User-data (Not aligned with the actual region where the cluster located).

3. Bottlerocket node would not join cluster and not able to SSH/SSM into the node for troubleshooting.

4. After taking volume snapshot and mount to another node, check the journal log. It's blocking by `pluto.service`

```bash
[root@ip-192-168-70-100 data-backup]# journalctl --file ./var/log/journal/ec278fedcd262b537e7eee9392040982/system.journal | grep pluto
Mar 05 05:03:58 [localhost](http://localhost/) pluto[1534]: Timed out retrieving private DNS name from EC2: deadline has elapsed
Mar 05 05:03:58 [localhost](http://localhost/) systemd[1]: pluto.service: Main process exited, code=exited, status=1/FAILURE
Mar 05 05:03:58 [localhost](http://localhost/) systemd[1]: pluto.service: Failed with result 'exit-code'.
```

**How to reproduce the problem:**

See above message.

***--= Edit =--***

After deep dive, I can see `pluto` plays a key role in generating Kubernetes corresponding parameter configurations during Bottlerocket startup. It involves interactions such as IMDS, AWS EC2 API and Kubernetes API as describe at GitHub page:

https://github.com/bottlerocket-os/bottlerocket-core-kit/tree/develop/sources/api/pluto

In the process, `pluto` will confirm the current EC2 Instance ID through IMDS when `generate_node_name()` is call:

https://github.com/bottlerocket-os/bottlerocket-core-kit/blob/develop/sources/api/pluto/src/main.rs#L464-L470

Next, send the EC2 Instance ID to `get_private_dn_name(region, &instance_id,...)` to confirm the `hostname` of the node:

https://github.com/bottlerocket-os/bottlerocket-core-kit/blob/develop/sources/api/pluto/src/main.rs#L473-L483

Among them, `get_private_dn_name()` will try repeatedly until the default timeout of `5 minutes`, which may cause the error to appear as you have observed:

https://github.com/bottlerocket-os/bottlerocket-core-kit/blob/develop/sources/api/pluto/src/ec2.rs#L43-L87

Here you can see that the key factors affecting the success or failure of `get_private_dn_name()` are the following parameters:

region, instance_id, http_proxy, no_proxy

If this step fails, the node may not be able to join the cluster.

Contributor guide

Open the contributing guide

Research direction

Start with sources/api/pluto/src/main.rs around generate_node_name() and sources/api/pluto/src/ec2.rs around get_private_dn_name(); inspect how region, instance_id, http_proxy, and no_proxy are handled. Reproduce the mismatched settings.aws.region case and review the resulting pluto journal logs. Done should mean the misconfiguration has a defined, verifiable startup outcome without breaking valid configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.