canonical / canonical/cloud-init
Boot deadlocks forever in network stage: wait_for_network's 'systemctl start systemd-networkd-wait-online' never returns (26.1-0ubuntu3~26.04.1, OpenStack/netplan)
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 1.1k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 18
Description
## Bug report
After upgrading an Ubuntu 26.04 (resolute) OpenStack VM from cloud-init `26.1-0ubuntu2` to the SRU `26.1-0ubuntu3~26.04.1`, the **first reboot never completes**: the network stage blocks forever inside `wait_for_network()` on `systemctl start systemd-networkd-wait-online.service`. `cloud-init-network.service` ships `TimeoutSec=0`, and `ssh.socket`/`docker.service` are ordered after cloud-init, so the host ends up answering ICMP with **zero listeners** — effectively bricked (only provider-console/disk surgery recovers it). The hang reproduced deterministically on every boot attempt (4/4), including after a full stop/start of the VM.
The same image with `26.1-0ubuntu2` (fresh VM, same cloud, same datasource) executes the identical call in **0.125 s** — systemd starts networkd alongside wait-online and boot proceeds.
## Environment
- Ubuntu 26.04 LTS (resolute) cloud image, OpenStack provider (`DMI: OpenStack Foundation OpenStack Nova`), ConfigDrive (`config-2`) present, netplan renderer, single NIC (`ens3`, DHCP4)
- Broken boot: cloud-init `26.1-0ubuntu3~26.04.1`, systemd `259.5-0ubuntu3.4`, kernel `7.0.0-30-generic` (all three landed in one `apt-get upgrade`/`dist-upgrade` three days before the first reboot — so I cannot fully isolate cloud-init vs. systemd; the behavioral delta is in this code path)
- Working boot (for comparison): same image, cloud-init `26.1-0ubuntu2`, systemd `259.5-0ubuntu3`, kernel `7.0.0-15-generic`
## Evidence (from the dead host's disk, post-mortem)
`/var/log/cloud-init.log` — these are the **final lines**; nothing follows for 28+ minutes until the VM was power-cycled:
```
main.py[DEBUG]: Will wait for network connectivity before continuing
stages.py[DEBUG]: Using distro class
activators.py[DEBUG]: Using selected activator: from priority: ['netplan', 'eni', 'network-manager', 'networkd']
subp.py[DEBUG]: Running command ['systemctl', 'is-enabled', 'NetworkManager.service'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['systemctl', 'start', 'systemd-networkd-wait-online.service'] with allowed return codes [0] (shell=False, capture=True)
```
Journal of the same boot: after `Starting cloud-init-network.service - Cloud-init: Network Stage...` at uptime ~6.5 s, **systemd logs zero further job transitions** for the rest of the boot (26+ min observed). `systemd-networkd.service` is never started in the main system (it ran only in the initrd, which got a DHCP lease fine and handed off). No `Ordering cycle`/`Deleting job` messages. The console shows only:
```
[*** ] Job cloud-init-network.service/start running (9min 2s / no limit)
```
So the start job for `systemd-networkd-wait-online.service` (`BindsTo=`/`After=systemd-networkd.service`) is enqueued but never becomes runnable, networkd's pulled-in job never runs either, and since the caller *is* the unit the rest of boot is ordered behind (`Before=sysinit.target`), the wait can never be satisfied — a deadlock that systemd's transaction cycle-checker cannot see because the jobs arrive in a second transaction.
On the working `26.1-0ubuntu2` boot, the same sequence logs:
```
subp.py[DEBUG]: Running command ['systemctl', 'start', 'systemd-networkd-wait-online.service'] with allowed return codes [0] (shell=False, capture=True)
performance.py[DEBUG]: Running ['systemctl', 'start', 'systemd-networkd-wait-online.service'] took 0.125 seconds
```
with the journal showing `Starting systemd-networkd.service...` immediately after `Starting cloud-init-network.service...`.
## Impact
A routine SRU upgrade turned into an unbootable production host on its next (unrelated, provider-initiated) reboot. Because `cloud-init-network.service` has `TimeoutSec=0`, there is no bound at all: sshd never starts, and the machine cannot be recovered without out-of-band access.
## Workarounds we validated
- Drop-in on `cloud-init-network.service` with `TimeoutStartSec=300` — the stage's job fails at the bound and boot proceeds (sshd/docker come up; `After=` is satisfied by a finished job). This seems worth shipping as a belt-and-braces default given the blast radius of `TimeoutSec=0`.
- Booting with `cloud-init=disabled` also recovers the host.
## Suggested angle
`wait_for_network()` issuing a **blocking** `systemctl start` from inside a unit that gates `sysinit.target` means any condition that keeps the started unit's job queued (rather than failing fast) converts to an infinite boot hang. Either the call needs its own timeout, or `cloud-init-network.service` should not ship `TimeoutSec=0`, or the wait should verify `systemd-networkd.service` is active/activatable before blocking on wait-online.
I no longer have the dead disk attached (provider reclaimed it), but I extracted full journals and logs for the broken and working boots before it went and can share more excerpts on request.
Contributor guide
Research direction
Start by tracing the wait_for_network() entry point and the cloud-init-network.service configuration, especially its TimeoutSec=0 setting. Reproduce the reported systemctl start systemd-networkd-wait-online.service behavior and compare the working and broken boot logs. Done means the network stage cannot leave boot blocked indefinitely and the relevant behavior is covered by a regression test or documented validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python, ubuntu
- Domain
- cloud, infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100