canonical / canonical/cloud-init

Oracle metadata fetch fails on IPv6-only Resolute instances

Open
#6,849 3 comments 0 reactions 0 assignees View on GitHub
bug incomplete
Dominant language
Python
Stars
3.8k
Forks
1.1k
Avg merge
2d 23h
Merged PRs (30d)
18

Description

### Description

Opening this issue to track a discussion and debug session regarding IPv6-only Oracle minimal instances.

On Ubuntu 26.04 (Resolute), cloud-init fails to fetch IMDS metadata. When comparing the boot behavior against Ubuntu 24.04 (Noble), we observed several differences in networking bring-up, timeouts, and early boot configuration.

### Environment
* **Cloud Provider:** Oracle Cloud Infrastructure (OCI)
* **Instance Type:** IPv6-only VM
* **OS Image:** Ubuntu 26.04 (Resolute) minimal image
* **cloud-init:** 26.1-0ubuntu2
* **Comparison Image:** Ubuntu 24.04 (Noble) minimal image

### Steps to Reproduce
1. Launch an IPv6-only Oracle Cloud instance using the Ubuntu 26.04 (Resolute) minimal image.
2. Allow the instance to boot.
3. Review `/var/log/cloud-init-output.log` and `/var/log/cloud-init.log`.

### Expected Behavior
`cloud-init` should successfully establish an ephemeral network connection (or use the early boot network configuration) to route to the IPv6 IMDS endpoint (`[fd00:c1::a9fe:a9fe]`) and successfully fetch instance metadata.

### Actual Behavior
`cloud-init` fails to fetch IMDS metadata. While `cloud-init` logs that the `enp0s5` interface was successfully brought up for IPv6 ephemeral networking, it remains unable to route to the IPv6 IMDS address. Additionally, the parallel IPv4 DHCP fallback throws an expected but noisy error, and the IPv6 IMDS retry loop crashes due to a bug setting the timeout to `0.0`.

---

### Observations & Logs

[resolute-cloud-init.tar.gz](https://github.com/user-attachments/files/26942906/cloud-init.tar.gz)

**1. Unroutable IMDS despite successful IPv6 bring-up (Resolute)**
During ephemeral network setup, `cloud-init` attempts to bring up IPv4 and IPv6 in parallel.
For IPv4, `dhcpcd` uses `--ipv4only` and expectedly times out on the IPv6-only instance, logging an error:
```text
2026-04-16 21:45:40,128 - dhcp.py[ERROR]: No valid DHCP lease configuration found in dhcpcd lease: ''
```
For IPv6, `cloud-init` logs that the ephemeral networking bring-up was actually successful:
```text
ephemeral.py[DEBUG]: Successfully brought up enp0s5 for ephemeral ipv6 networking
```
Despite IPv6 networking being up, `cloud-init` is still unable to route to the IPv6 IMDS address at that point in time.

**2. Timeout Bug in `url_helper.py` preventing retries**
During the retries for the IPv6 IMDS, there is a separate bug where the timeout drops to `0.0`.

```text
url_helper.py[DEBUG]: Calling 'url' failed [29/30s]: unexpected error [Attempted to set connect timeout to 0.0, but the timeout cannot be set to a value less than or equal to 0.]
```

**3. Noble vs. Resolute Network Bring-up Differences**

[noble-cloud-init.tar.gz](https://github.com/user-attachments/files/26942907/noble-cloud-init.tar.gz)

* **Noble (Works):** Initially shows no carrier on the primary NIC (`Interface has no carrier: enp0s5`), causing `cloud-init` to ignore it in the `init-local` timeframe. `dhcpcd` tries IPv4 and times out after sleep retry loops. After the IPv4 failure, IPv6 is successfully brought up, and it successfully fetches IMDS (`DataSourceOracle.py[DEBUG]: Successfully fetched vnics metadata from IMDS at: http://[fd00:c1::a9fe:a9fe]/opc/v2/vnics/`).
* **Resolute (Fails):** The carrier is present much earlier in `systemd-networkd` logs, even before `cloud-init-main` or `cloud-init-local` starts:
```text
Apr 16 21:45:06.476436 localhost systemd-networkd[383]: enp0s5: Configuring with /run/systemd/network/71-default.network.
Apr 16 21:45:06.477202 localhost systemd-networkd[383]: enp0s5: Link UP
Apr 16 21:45:06.477579 localhost systemd-networkd[383]: enp0s5: Gained carrier
```

---

### Action Items & Suspected Root Causes (from @blackboxsw )

Based on the logs and conversation we had, Chad noted the following items that need to be addressed/investigated:

1. **Reduce dhcp.py ERROR log level:** It is okay for `dhcpcd --ipv4only` to fail because IPv4 and IPv6 ephemeral network setups are attempted in parallel and IPv6 can be successful. The log level should be reduced from `ERROR`.
2. **Fix url_helper.py timeout bug:** Fix the logic to avoid setting the timeout to `0.0` instead of `5.0` seconds during retries of IPv6 IMDS.
3. **Investigate dracut / iSCSI differences:** Chad suspects the missing routable connection to IMDS might be related to `dracut` behavior differences in Resolute.
* `cloud-init` early boot detection previously relied on the presence of `initramfs` files in `/run/net-*.conf` and `/run/net6-*.conf`.
* Because Resolute is `dracut`-based, these files are presumed absent, meaning network config info from the `initramfs` setup is missing.
* The kernel cmdline in the Resolute case points to all things `dracut` and ISCSI, which appears to be where the problem lies.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.