canonical / canonical/cloud-init

[bug]: EC2 datasource failing on first boot due to missing NICs

Open
#6,697 12 comments 3 reactions 0 assignees View on GitHub
bug incomplete
Dominant language
Python
Stars
3.8k
Forks
1.1k
Avg merge
2d 23h
Merged PRs (30d)
18

Description

# Bug report
On Ubuntu 24.04 Minimal EC2 AMIs, cloud-init may fail to retrieve EC2 metadata and userdata on the first boot for certain instance types (notably hpc7a.*, every time on hpc7a.96xlarge). This results in SSH keys from user-data not being applied and prevents SSH access until the instance is rebooted. On the first boot, the data source fallsback to DataSourceNone. On reboot, the data source is set correctly, but there are still NIC errors and a status degraded.

After a reboot, cloud-init successfully retrieves metadata and applies the correct SSH keys, allowing login. The issue appears to be caused by a race condition where no eligible network interfaces are present during the init-local stage when the EC2 datasource attempts metadata discovery.

This manifests as recoverable errors in `cloud-init status --long`, even though the instance eventually boots.

## Steps to reproduce the problem
1. Launch an Ubuntu 24.04 Noble Minimal EC2 AMI
2. Use an EC2 instance type where network interfaces may not be immediately available at early boot (e.g. hpc7a.96xlarge reproduces consistently)
3. Wait several minutes after the instance reaches running, then attempt to SSH into the instance
```
# ssh with public ip
ssh ubuntu@

# After a while, it will most likely timeout and you will get this error:
ssh: connect to host port 22: Connection timed out

# and if it doesn't timeout, you will get this error
ubuntu@: Permission denied (publickey).
```
4. Reboot the instance
5. After reboot, SSH access should work as expected

## Observed behavior
There is a datasource error (fallback to DataSourceNone) and cloud-init fails to retrieve the user metadata.

```
[ 358.343691] cloud-init[2407]: 2026-01-30 15:39:39,277 - DataSourceEc2.py[ERROR]: Unable to get metadata
Stopping systemd-networkd.service - Network Configuration...
.
.
.
ci-info: no authorized SSH keys fingerprints found for user ubuntu.
.
.
.
[ 363.690888] cloud-init[3073]: Cloud-init v. 25.2-0ubuntu1~24.04.1 finished at Fri, 30 Jan 2026 15:39:44 +0000. Datasource DataSourceNone. Up 363.68 seconds
[ 363.693080] cloud-init[3073]: 2026-01-30 15:39:44,637 - cc_final_message.py[WARNING]: Used fallback datasource
```

## Expected behavior
- cloud-init should successfully retrieve metadata and userdata on first boot, without requiring a reboot.
- SSH keys provided via EC2 key-pair or user-data should be applied reliably on initial instance launch.

## Environment details
- Cloud-init version: cloud-init 25.2-0ubuntu1~24.04.1
- Operating System Distribution: Ubuntu 24.04 LTS (Noble) Minimal AMI
- Cloud provider, platform or installer type: Amazon EC2, hpc7a.* (notably hpc7a.96xlarge) instances

## cloud-init logs

On the first boot , relevant logs show:
```
.
.
.
[FAILED] Failed to start systemd-networkd-w… Wait for Network to be Configured.

See 'systemctl status systemd-networkd-wait-online.service' for details.

Starting cloud-init.service - Cloud-init: Network Stage...

[ 126.271452] cloud-init[2407]: Cloud-init v. 25.2-0ubuntu1~24.04.1 running 'init' at Fri, 30 Jan 2026 15:35:47 +0000. Up 126.25 seconds.
[ 126.293820] cloud-init[2407]: ci-info: ++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[ 126.295452] cloud-init[2407]: ci-info: +---------+-------+-----------+-----------+-------+-------------------+
[ 126.296909] cloud-init[2407]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 126.311334] cloud-init[2407]: ci-info: +---------+-------+-----------+-----------+-------+-------------------+
[ 126.312778] cloud-init[2407]: ci-info: | enp34s0 | False | . | . | . | 06:92:08:52:f5:89 |
[ 126.314263] cloud-init[2407]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
[ 126.315732] cloud-init[2407]: ci-info: | lo | True | ::1/128 | . | host | . |
[ 126.317201] cloud-init[2407]: ci-info: +---------+-------+-----------+-----------+-------+-------------------+
[ 126.318690] cloud-init[2407]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[ 126.319953] cloud-init[2407]: ci-info: +-------+-------------+---------+-----------+-------+
[ 126.321208] cloud-init[2407]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[ 126.322470] cloud-init[2407]: ci-info: +-------+-------------+---------+-----------+-------+
[ 126.323731] cloud-init[2407]: ci-info: +-------+-------------+---------+-----------+-------+
[ 126.455370] cloud-init[2407]: 2026-01-30 15:35:47,402 - url_helper.py[WARNING]: Calling 'http://[fd00:ec2::254]/latest/api/token' failed [0/240s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))]
[ 127.608564] cloud-init[2407]: 2026-01-30 15:35:48,554 - url_helper.py[WARNING]: Calling 'http://[fd00:ec2::254]/latest/api/token' failed [1/240s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))]
.
.
.

[ 358.343691] cloud-init[2407]: 2026-01-30 15:39:39,277 - DataSourceEc2.py[ERROR]: Unable to get metadata
Stopping systemd-networkd.service - Network Configuration...

[ OK ] Stopped systemd-networkd.service - Network Configuration.

Starting systemd-networkd.service - Network Configuration...

[ OK ] Started systemd-networkd.service - Network Configuration.

[ 359.565442] cloud-init[2407]: Generating public/private rsa key pair.
[ 359.566557] cloud-init[2407]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key
[ 359.567816] cloud-init[2407]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
[ 359.580983] cloud-init[2407]: The key fingerprint is:
[ 359.581727] cloud-init[2407]: SHA256:WOk2SyyWGHZPJ9Yg9oB4XqxqksD1sXji6Q5/Vi4RKS4 root@ubuntu
[ 359.582998] cloud-init[2407]: The key's randomart image is:
[ 359.585053] cloud-init[2407]: +---[RSA 3072]----+
[ 359.585727] cloud-init[2407]: | . o+ . |
[ 359.586406] cloud-init[2407]: | ..ooo+ + |
[ 359.587114] cloud-init[2407]: |. .o=o= B o |
[ 359.587819] cloud-init[2407]: |.. =oX X o |
[ 359.600997] cloud-init[2407]: |..o.* * S |
[ OK ] Finished cloud-init.service - Cloud-init: Network Stage.
.
.
.

[ 363.582117] cloud-init[3073]: Cloud-init v. 25.2-0ubuntu1~24.04.1 running 'modules:final' at Fri, 30 Jan 2026 15:39:44 +0000. Up 363.56 seconds.
ci-info: no authorized SSH keys fingerprints found for user ubuntu.
<14>Jan 30 15:39:44 cloud-init: #############################################################
<14>Jan 30 15:39:44 cloud-init: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Jan 30 15:39:44 cloud-init: 256 SHA256:xoOe09zhAt7z01JXUhNcvj7p9DTJ7WyJeJeJVEwUIFo root@ubuntu (ECDSA)
<14>Jan 30 15:39:44 cloud-init: 256 SHA256:duy4AbfTRrnzFIoUXQpCE/xzkj/HBxslkmlTb7ZAvxI root@ubuntu (ED25519)
<14>Jan 30 15:39:44 cloud-init: 3072 SHA256:WOk2SyyWGHZPJ9Yg9oB4XqxqksD1sXji6Q5/Vi4RKS4 root@ubuntu (RSA)
<14>Jan 30 15:39:44 cloud-init: -----END SSH HOST KEY FINGERPRINTS-----
<14>Jan 30 15:39:44 cloud-init: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAfMYpmVwZGz0Qzn7RZOcxOgcEKk/J+2XIhbUcrXTF8UPID1v8HnCmqxYWro8qXOAw9kvBtdqnFBYyzvMtpqIls= root@ubuntu
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKY5AXx3RlaCdg2TusRbvBRnxDtv9OEtiz9cj6aDOA0c root@ubuntu
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2Fngep4u0IWa+vxhnrAQllVSpq/22pCMqkylLh+cMelbTGaedhLbTeqeF5pHSudr0CNg1ZRukrP8Y5+ST3nOxHug3KW3vT3fNHOatvOu90Ys3bkaFjROTfylz65EB4QTfNqKFCIUvu8O/e8lQyAbT9NqcP/rW9aQ067DVBZtl66EyhNy073Hp368iXHQ6vR+XelRPXu3QyX3Kyr2mSlg2hMsEYCOTg63n8G5WTaxZzqOOVgZM2RqrE85owgw/+e0RGh3TF1p3PIqqqr7GpIF3LGcxVrFtrjK5XF1PuOrPyHzFkP1gqB+crfI90eJkVPJaxBf1Qy8T7NqXjCpDc5nFbcG8WFUZBIImcWd4TcuvPlYi398z3Tn3IjZlg4LzmiBJbKSR8Skuep8yQdo4BX1DWRhOPEsZ3vnFJOIIuQ9xeJZXQVH26sIvX5uzUo8DbY6+mxnnMZiXUCXGXy8Eo1BjwB47jZGxE3P1gePSBNayKd+HkaxpUP5VFSg9ioM1UO0= root@ubuntu
-----END SSH HOST KEY KEYS-----
[ 363.690888] cloud-init[3073]: Cloud-init v. 25.2-0ubuntu1~24.04.1 finished at Fri, 30 Jan 2026 15:39:44 +0000. Datasource DataSourceNone. Up 363.68 seconds
[ 363.693080] cloud-init[3073]: 2026-01-30 15:39:44,637 - cc_final_message.py[WARNING]: Used fallback datasource
[ OK ] Finished cloud-final.service - Cloud-init: Final Stage.

[ OK ] Reached target cloud-init.target - Cloud-init target.

Ubuntu 24.04.3 LTS ubuntu ttyS0
```

Note these errors especially (from the log above):
```
[ 358.343691] cloud-init[2407]: 2026-01-30 15:39:39,277 - DataSourceEc2.py[ERROR]: Unable to get metadata
Stopping systemd-networkd.service - Network Configuration...
.
.
.
ci-info: no authorized SSH keys fingerprints found for user ubuntu.
.
.
.
[ 363.690888] cloud-init[3073]: Cloud-init v. 25.2-0ubuntu1~24.04.1 finished at Fri, 30 Jan 2026 15:39:44 +0000. Datasource DataSourceNone. Up 363.68 seconds
[ 363.693080] cloud-init[3073]: 2026-01-30 15:39:44,637 - cc_final_message.py[WARNING]: Used fallback datasource
```

When I try to SSH, I get
```
ubuntu@seg-aws:~$ ssh -i ~/goldberl.pem ubuntu@
ubuntu@18.118.46.52: Permission denied (publickey).
```

This is `ip l` on the first boot and after reboot
```
ubuntu@ip-172-31-23-161:~$ ip l
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp34s0: mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 06:92:08:52:f5:89 brd ff:ff:ff:ff:ff:ff
```
`cloud-init status --long` (on first boot)
```
ubuntu@ubuntu:~$ cloud-init status --long
status: done
extended_status: degraded done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:06:08 +0000
detail: DataSourceNone
errors: []
recoverable_errors:
ERROR:
- Unable to get response from urls: ['http://169.254.169.254/latest/api/token', 'http://[fd00:ec2::254]/latest/api/token']
- Unable to get metadata
- The instance must have at least one eligible NIC
- The instance must have at least one eligible NIC
- The instance must have at least one eligible NIC
WARNING:
- Calling 'http://[fd00:ec2::254]/latest/api/token' failed [0/240s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))]
.
.
.
- Calling 'http://[fd00:ec2::254]/latest/api/token' failed [232/240s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))]
- IMDS's HTTP endpoint is probably disabled
- Used fallback datasource
```

Also `cloud-init status --long` (on reboot)
```
ubuntu@ip-172-31-23-161:~$ cloud-init status --long
status: done
extended_status: degraded done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:00:08 +0000
detail: DataSourceEc2
errors: []
recoverable_errors:
ERROR:
- The instance must have at least one eligible NIC
- The instance must have at least one eligible NIC
- The instance must have at least one eligible NIC
- The instance must have at least one eligible NIC
WARNING:
- Calling 'http://[fd00:ec2::254]/latest/api/token' failed [0/240s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.