bluerobotics / bluerobotics/BlueOS
bug: dhcpcd deletes the shared 192.168.2.0/24 route on lease loss, orphaning cable-guy's static eth0 address (ENETUNREACH, 40 min comms loss underway)
- Dominant language
- Vue
- Stars
- 453
- Forks
- 151
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 174
Description
### Bug description
| | |
|---|---|
| BlueOS | `1.5.0-beta.39-0-g7adad3f6` |
| Hardware | Raspberry Pi 4 Model B Rev 1.5 |
| OS / kernel | Debian 12 (bookworm), `5.10.33-v7l+` |
| Vehicle | BlueBoat, ArduRover 4.7.0 (`1511f271`) |
| `eth0` config | static `192.168.2.12/24` (cable-guy, `AddressMode.Unmanaged`) **and** host `dhcpcd` leasing `192.168.2.143/24` — both on the same subnet |
| MAVLink endpoints | UDP clients to `192.168.2.1:14550` and `192.168.2.128:14550` |
| Date of incident | 2026-07-27 / 2026-07-28, all times UTC |
**Summary**
`eth0` carries two IPv4 configurations on the same subnet: the static `192.168.2.12/24` that cable-guy applies, and a `192.168.2.143/24` lease that the host's `dhcpcd` obtains. They share one connected route, `192.168.2.0/24`, and `dhcpcd` believes it owns it.
A physical-layer fault caused 38 `eth0` carrier losses in under an hour. Every time `dhcpcd` lost its lease it also ran `eth0: deleting route to 192.168.2.0/24`. At **23:37:32 it deleted that route for the last time and never re-added it**. The static `192.168.2.12` remained on the interface, but with no route to its own subnet every outbound packet to `192.168.2.x` failed with `ENETUNREACH` — for 40 minutes, including the final 27½ minutes during which the carrier was up and stable.
`mavlink-server` logged 11,220 send failures on this boot, 11,085 of them to `192.168.2.1:14550` and `192.168.2.128:14550`, all identical:
```
Failed to send message: Os { code: 101, kind: NetworkUnreachable, message: "Network unreachable" }
```
From the operator's point of view `192.168.2.12` was dead while every other device on the boat network still responded. The vehicle was underway, station-keeping in GUIDED at a commanded position. Recovery required physically reaching it and power-cycling it.
**Evidence that the address survived and the route did not**
- `avahi-daemon` withdrew and re-registered `192.168.2.143` on every single flap. It registered `192.168.2.12` once at 21:28:30 and **never withdrew it** — so the static address was on the interface for the entire incident.
- `ENETUNREACH` (errno 101) is what the kernel returns when there is no route to the destination network. A missing address or a dead link produce different errors.
- `dhcpcd` logged `eth0: adding route to 192.168.2.0/24` followed one second later by `eth0: pid 679 deleted route to 192.168.2.0/24`, twice (22:47:01/02 and 22:54:07). That is the two configurations fighting over the same prefix route.
- After the carrier stabilised at 23:50:09, `dhcpcd` logged exactly one solicitation (23:50:10) and then nothing at all for 27½ minutes. Nothing else in the stack noticed or intervened.
**Independent confirmation from a second process**
A separate logger running on the same Pi polls the vehicle's Mikrotik radio over `192.168.2.x` at ~1.5 s intervals. It went blind at exactly the same moments as `mavlink-server` — each of its long outages begins 2–5 s after a `dhcpcd` route deletion and ends 2–14 s after the route is restored — while its position column, read locally over MAVLink, kept updating normally throughout. The Pi itself was healthy; only the path to `192.168.2.0/24` was gone.
That log also rules out the radio as the cause of the 40-minute outage:
- The vehicle drifted from 531 m to 208 m during the blackout — steadily *closer* to the topside station and into stronger signal — and never recovered.
- Measured median SNR by range on the same afternoon: 45 dB at 100–199 m, 41 dB at 200–299 m, 37 dB at 300–399 m. The blackout was spent entirely inside the range band that had been carrying full-rate traffic all day.
- At 23:07:02 the kernel logged `Link is Down`, the carrier returned at 23:07:05, and the radio answered a poll at 23:07:05 with live data (27 dB SNR, 18 Mbps). A RouterOS device that had just rebooted could not answer three seconds later — the radio never went down. Polls only began failing at 23:07:10, the instant the lease expired.
- When the link was finally re-tested at 00:21:24 from 168 m, it associated instantly at 43 dB and 72.2 Mbps.
**What actually restored service**
Not DHCP. On the following boot `dhcpcd` solicited from 00:17:18 and gave up with `timed out` at 00:17:44; no lease was ever granted again. Comms returned at **00:18:58, when cable-guy applied the static `192.168.2.12`** — and this time there was no DHCP client on the same subnet to delete its prefix route. That is the entire difference between the working and non-working states.
**This is not a one-off**
The same signature appears on a previous outing: 2026-07-23, 22:12:15–22:30:59, 6,078 `ENETUNREACH` failures split evenly between `192.168.2.1` and `192.168.2.128`.
**Question for maintainers**
Does cable-guy add the static address in a way that sets `IFA_F_NOPREFIXROUTE` (or otherwise suppresses the kernel's automatic prefix route)? If so, `dhcpcd`'s route is the *only* route for `192.168.2.0/24` and deleting it is immediately fatal, which matches everything observed here. If the kernel prefix route does normally exist, then `dhcpcd` is deleting a route it did not create, which is also a problem.
### Steps to reproduce
1. Configure `eth0` with a cable-guy static address on a subnet that also has a DHCP server (the stock `192.168.2.0/24` tether subnet).
2. Let the host `dhcpcd` obtain a lease on that same subnet.
3. Flap the link repeatedly — unplug/replug ethernet, or use a marginal Wi-Fi bridge — until `dhcpcd` cannot complete a handshake.
4. Restore the link and leave it up. The interface still holds the static address, but `ip route` no longer has `192.168.2.0/24 dev eth0`, and everything on that subnet is unreachable indefinitely.
**Timeline (UTC)**
| Time | Event | Source |
|---|---|---|
| 21:17:01 | Boot | journal |
| 21:28:30 | avahi registers `192.168.2.12` on eth0 — never withdrawn for the rest of the boot | journal |
| 22:47:01 | First DHCP lease `192.168.2.143`; `adding route to 192.168.2.0/24` | journal |
| 22:47:02 | `pid 679 deleted route to 192.168.2.0/24` — static and DHCP configs collide | journal |
| 22:53:02 | First eth0 carrier loss | journal |
| 22:59:17 | First of 17 Pi 5 V under-voltage warnings | journal |
| 23:07:02 | Carrier loss; `mavlink-server` begins failing with `ENETUNREACH` | journal + mavlink-server |
| 23:07:05 | Carrier back; Mikrotik still answering with live link data — radio never went down | mikrotik log |
| 23:07:10 | Lease expires; subnet becomes unreachable | journal |
| 23:07:16 | ArduPilot declares GCS failsafe | dataflash |
| 23:10:55 | New lease, `adding route to 192.168.2.0/24` → 23:10:58 failsafe cleared | journal + dataflash |
| 23:22:10 / 23:28:31 | Two more `deleting route` / `adding route` cycles; both self-recovered | journal |
| 23:36:12 | `deleting route`; start of a burst of 33 carrier drops in 14 minutes | journal |
| 23:37:16 | Last successful DHCP lease; route re-added | journal |
| 23:37:32 | **`eth0: deleting route to 192.168.2.0/24` — never re-added** | journal |
| 23:37:52 | GCS failsafe declared; never cleared again | dataflash |
| 23:50:09 | `eth0: carrier acquired` — link stable from here to the end | journal |
| 23:50:10 | One `soliciting a DHCP lease`, then no eth0 activity logged at all | journal |
| 23:50:10 – 00:17:44 | 27½ min: carrier up, static address present, no route, no recovery | journal |
| 00:17:44 | Pi hard-resets (no shutdown sequence in the journal) | both |
| 00:17:18 – 00:17:44 | Next boot: `dhcpcd` solicits, then `timed out`. DHCP never works again | journal |
| 00:18:58 | cable-guy applies static `192.168.2.12`; comms restored | journal |
### Primary pain point(s)
An autonomous surface vehicle underway at sea was unreachable for 40 minutes, while its radio, its autopilot and its towfish payload all continued to function normally. Recovery required physically reaching the vehicle.
**Expected behaviour**
- A transient carrier loss causes a brief packet gap, not the removal of the interface's routing.
- A DHCP client must not be able to delete the connected route belonging to a statically configured address on the same interface.
- If the primary interface has carrier but cannot reach its own subnet for more than a few seconds, something escalates and repairs it. A watchdog that only checks "is the address present?" will not catch this — the address *was* present.
- A MAVLink endpoint unreachable for minutes is surfaced as a health condition, not only as repeated log lines.
**Actual behaviour**
None of the above. The interface sat with carrier, with its static address, and with no route to its own subnet, indefinitely.
**Possible mitigations**
- Do not run `dhcpcd` on an interface that cable-guy has statically configured on the same subnet — either exclude it in `dhcpcd.conf` via `denyinterfaces`, or have cable-guy stop the DHCP client when it applies a static address.
- Ensure the static address keeps its own kernel prefix route independent of any DHCP lease.
- Extend the cable-guy watchdog to verify *routability* (address **and** a matching route, or an actual ARP/ping to the gateway) rather than address presence alone, and to re-assert the static configuration when that check fails.
### Additional context
Logs: https://drive.google.com/drive/folders/1tqOLXCzkcC6oYky4V-z25GpODL4YBcQT?usp=sharing
The physical-layer fault that triggered this (38 carrier drops, plus 17 Pi 5 V under-voltage warnings) is a problem on our vehicle and not a BlueOS bug. The 40-minute unrecoverable outage that a 3-second carrier glitch was able to cause is the part being reported here.
### Prerequisites
- [x] I have checked to make sure that a similar request has not already been filed or fixed.
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file or test is named; start by reproducing the eth0 and dhcpcd lease-loss sequence from the listed steps, then trace cable-guy's static-address setup and route handling. Done means a DHCP lease loss cannot remove the route needed by the static address, or the watchdog detects and restores routability after the carrier returns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, raspberry-pi
- Domain
- embedded-iot, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100