mwan3: Potential race condition between mwan3 and tailscale

Maintainer: @feckert
Environment:

• x86/64 / OpenWrt 23.05.2 r23630-842932a63d / LuCI openwrt-23.05 branch git-23.306.39416-c86c256
• mwan3 2.11.8-1
• tailscale 1.56.1-2

Description:

We have set up mwan3 and tailscale in accordance with our specific requirements, and have included the configuration details for your reference.

However, we are encountering an issue where the routing from an internal subnet, which is behind OpenWRT, to the subnets behind tailscale is inconsistent upon rebooting the appliance. Sometimes it functions correctly, while at other times it does not. We suspect that this inconsistency may be due to a race condition between mwan3 and tailscale. Interestingly, once we restart tailscale, the system begins to operate as expected without any hitches.

We are currently implementing a temporary workaround, which involves introducing a delay before starting tailscale. However, this is intended to be a short-term solution.

Here are the conditions before restarting tailscale, when we are unable to connect to the tailscale mesh:

Priority	Rule
0	from all lookup local
1001	from all iif eth1 lookup 1
1003	from all iif eth2 lookup 3
2001	from all fwmark 0x100/0x3f00 lookup 1
2003	from all fwmark 0x300/0x3f00 lookup 3
2061	from all fwmark 0x3d00/0x3f00 blackhole
2062	from all fwmark 0x3e00/0x3f00 unreachable
3001	from all fwmark 0x100/0x3f00 unreachable
3003	from all fwmark 0x300/0x3f00 unreachable
5210	from all fwmark 0x80000/0xff0000 lookup main
5230	from all fwmark 0x80000/0xff0000 lookup default
5250	from all fwmark 0x80000/0xff0000 unreachable
5270	from all lookup 52
32766	from all lookup main
32767	from all lookup default

After restarting tailscale (or when it initially works properly):

Priority	Rule
0	from all lookup local
1001	from all iif eth1 lookup 1
1003	from all iif eth2 lookup 3
1310	from all fwmark 0x80000/0xff0000 lookup main
1330	from all fwmark 0x80000/0xff0000 lookup default
1350	from all fwmark 0x80000/0xff0000 unreachable
1370	from all lookup 52
2001	from all fwmark 0x100/0x3f00 lookup 1
2003	from all fwmark 0x300/0x3f00 lookup 3
2061	from all fwmark 0x3d00/0x3f00 blackhole
2062	from all fwmark 0x3e00/0x3f00 unreachable
3001	from all fwmark 0x100/0x3f00 unreachable
3003	from all fwmark 0x300/0x3f00 unreachable
32766	from all lookup main
32767	from all lookup default

Here are the respective configuration files:
firewall-config.txt
network-config.txt
tailscale-config.txt
mwan3-config.txt
tailscale-up-command.txt

Here are the boot logs for both an example where it was working and one where it failed:
boot-log-failure.txt
boot-log-success.txt

Any help is highly appreciated.

Start by comparing boot-log-failure.txt with boot-log-success.txt, then review the attached firewall, network, mwan3, and tailscale configuration files alongside the two routing-rule snapshots. Determine why the routing state differs after reboot and restarting tailscale, and document or implement a fix that makes the expected routing state consistent without the startup delay workaround.