canonical / canonical/cloud-init

Boot deadlocks forever in network stage: wait_for_network's 'systemctl start systemd-networkd-wait-online' never returns (26.1-0ubuntu3~26.04.1, OpenStack/netplan)

Open
#7,052 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.8k
Forks
1.1k
Avg merge
2d 23h
Merged PRs (30d)
18

Description

## Bug report

After upgrading an Ubuntu 26.04 (resolute) OpenStack VM from cloud-init `26.1-0ubuntu2` to the SRU `26.1-0ubuntu3~26.04.1`, the **first reboot never completes**: the network stage blocks forever inside `wait_for_network()` on `systemctl start systemd-networkd-wait-online.service`. `cloud-init-network.service` ships `TimeoutSec=0`, and `ssh.socket`/`docker.service` are ordered after cloud-init, so the host ends up answering ICMP with **zero listeners** — effectively bricked (only provider-console/disk surgery recovers it). The hang reproduced deterministically on every boot attempt (4/4), including after a full stop/start of the VM.

The same image with `26.1-0ubuntu2` (fresh VM, same cloud, same datasource) executes the identical call in **0.125 s** — systemd starts networkd alongside wait-online and boot proceeds.

## Environment

- Ubuntu 26.04 LTS (resolute) cloud image, OpenStack provider (`DMI: OpenStack Foundation OpenStack Nova`), ConfigDrive (`config-2`) present, netplan renderer, single NIC (`ens3`, DHCP4)
- Broken boot: cloud-init `26.1-0ubuntu3~26.04.1`, systemd `259.5-0ubuntu3.4`, kernel `7.0.0-30-generic` (all three landed in one `apt-get upgrade`/`dist-upgrade` three days before the first reboot — so I cannot fully isolate cloud-init vs. systemd; the behavioral delta is in this code path)
- Working boot (for comparison): same image, cloud-init `26.1-0ubuntu2`, systemd `259.5-0ubuntu3`, kernel `7.0.0-15-generic`

## Evidence (from the dead host's disk, post-mortem)

`/var/log/cloud-init.log` — these are the **final lines**; nothing follows for 28+ minutes until the VM was power-cycled:

```
main.py[DEBUG]: Will wait for network connectivity before continuing
stages.py[DEBUG]: Using distro class
activators.py[DEBUG]: Using selected activator: from priority: ['netplan', 'eni', 'network-manager', 'networkd']
subp.py[DEBUG]: Running command ['systemctl', 'is-enabled', 'NetworkManager.service'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['systemctl', 'start', 'systemd-networkd-wait-online.service'] with allowed return codes [0] (shell=False, capture=True)
```

Journal of the same boot: after `Starting cloud-init-network.service - Cloud-init: Network Stage...` at uptime ~6.5 s, **systemd logs zero further job transitions** for the rest of the boot (26+ min observed). `systemd-networkd.service` is never started in the main system (it ran only in the initrd, which got a DHCP lease fine and handed off). No `Ordering cycle`/`Deleting job` messages. The console shows only:

```
[*** ] Job cloud-init-network.service/start running (9min 2s / no limit)
```

So the start job for `systemd-networkd-wait-online.service` (`BindsTo=`/`After=systemd-networkd.service`) is enqueued but never becomes runnable, networkd's pulled-in job never runs either, and since the caller *is* the unit the rest of boot is ordered behind (`Before=sysinit.target`), the wait can never be satisfied — a deadlock that systemd's transaction cycle-checker cannot see because the jobs arrive in a second transaction.

On the working `26.1-0ubuntu2` boot, the same sequence logs:

```
subp.py[DEBUG]: Running command ['systemctl', 'start', 'systemd-networkd-wait-online.service'] with allowed return codes [0] (shell=False, capture=True)
performance.py[DEBUG]: Running ['systemctl', 'start', 'systemd-networkd-wait-online.service'] took 0.125 seconds
```

with the journal showing `Starting systemd-networkd.service...` immediately after `Starting cloud-init-network.service...`.

## Impact

A routine SRU upgrade turned into an unbootable production host on its next (unrelated, provider-initiated) reboot. Because `cloud-init-network.service` has `TimeoutSec=0`, there is no bound at all: sshd never starts, and the machine cannot be recovered without out-of-band access.

## Workarounds we validated

- Drop-in on `cloud-init-network.service` with `TimeoutStartSec=300` — the stage's job fails at the bound and boot proceeds (sshd/docker come up; `After=` is satisfied by a finished job). This seems worth shipping as a belt-and-braces default given the blast radius of `TimeoutSec=0`.
- Booting with `cloud-init=disabled` also recovers the host.

## Suggested angle

`wait_for_network()` issuing a **blocking** `systemctl start` from inside a unit that gates `sysinit.target` means any condition that keeps the started unit's job queued (rather than failing fast) converts to an infinite boot hang. Either the call needs its own timeout, or `cloud-init-network.service` should not ship `TimeoutSec=0`, or the wait should verify `systemd-networkd.service` is active/activatable before blocking on wait-online.

I no longer have the dead disk attached (provider reclaimed it), but I extracted full journals and logs for the broken and working boots before it went and can share more excerpts on request.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the wait_for_network() entry point and the cloud-init-network.service configuration, especially its TimeoutSec=0 setting. Reproduce the reported systemctl start systemd-networkd-wait-online.service behavior and compare the working and broken boot logs. Done means the network stage cannot leave boot blocked indefinitely and the relevant behavior is covered by a regression test or documented validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python, ubuntu
Domain
cloud, infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.