cross-vm network connectivity issue when using socket_vmnet
- Dominant language
- Go
- Stars
- 21.9k
- Forks
- 957
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 53
Description
### Description
I have frequently hit this issue when running VMs with `lima:shared` networking (socker_vmnet).
When this happens, the cross-vm traffic completely stalls. I am happy to provide further details that would help get to the bottom of this issue.
```
Aug 15 00:53:22 lima-murphy-cli-01 kernel: ------------[ cut here ]------------
Aug 15 00:53:22 lima-murphy-cli-01 kernel: NETDEV WATCHDOG: lima0 (virtio_net): transmit queue 0 timed out 6080 ms
Aug 15 00:53:22 lima-murphy-cli-01 kernel: WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x2a8/0x2b0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: Modules linked in: xt_multiport xt_nat xt_mark xt_comment veth tls xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink ip6table_nat ip6table_filter ip6_tables iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype iptable_filter ip_tables br_netfilter bridge stp llc overlay rfkill isofs vfat fat virtio_gpu virtio_dma_buf drm_shmem_helper drm_kms_helper fb_sys_fop
s syscopyarea sysfillrect sysimgblt virtio_input drm fuse xfs libcrc32c sr_mod cdrom sg crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_blk virtio_console virtio_net net_failover virtio_scsi failover virtio_mmio sunrpc dm_mirror dm_region_hash dm_log dm_mod
Aug 15 00:53:22 lima-murphy-cli-01 kernel: CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.14.0-427.13.1.el9_4.aarch64 #1
Aug 15 00:53:22 lima-murphy-cli-01 kernel: Hardware name: QEMU QEMU Virtual Machine, BIOS edk2-stable202402-prebuilt.qemu.org 02/14/2024
Aug 15 00:53:22 lima-murphy-cli-01 kernel: pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
Aug 15 00:53:22 lima-murphy-cli-01 kernel: pc : dev_watchdog+0x2a8/0x2b0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: lr : dev_watchdog+0x2a8/0x2b0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: sp : ffff800080013de0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x29: ffff800080013de0 x28: ffffc0360445a9c0 x27: ffff800080013e88
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x26: ffffc036052f2008 x25: 00000000000017c0 x24: ffffc03605757000
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x23: 0000000000000000 x22: ffff162d080e03dc x21: ffff162d080e0000
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x20: ffff162d080e0488 x19: ffff162d00412a00 x18: ffffffffffffffff
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x17: ffff55f73a0d6000 x16: ffff800080010000 x15: ffffc03605d443f9
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x14: ffffffffffffffff x13: ffffc03605d443f0 x12: 696d736e61727420
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x11: 656820747563205b x10: 0000000000000028 x9 : ffffc03603a94680
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x8 : 000000000a736d20 x7 : 2074756f2064656d x6 : 000000000a736d20
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x5 : ffff162d3f3cb950 x4 : 0000000000001000 x3 : 0000000000000040
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff162d002f6900
Aug 15 00:53:22 lima-murphy-cli-01 kernel: Call trace:
Aug 15 00:53:22 lima-murphy-cli-01 kernel: dev_watchdog+0x2a8/0x2b0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: call_timer_fn+0x38/0x1d0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: __run_timers.part.0+0x22c/0x300
Aug 15 00:53:22 lima-murphy-cli-01 kernel: run_timer_softirq+0x40/0x74
Aug 15 00:53:22 lima-murphy-cli-01 kernel: __do_softirq+0x118/0x3d8
Aug 15 00:53:22 lima-murphy-cli-01 kernel: ____do_softirq+0x14/0x20
Aug 15 00:53:22 lima-murphy-cli-01 kernel: call_on_irq_stack+0x24/0x30
Aug 15 00:53:22 lima-murphy-cli-01 kernel: do_softirq_own_stack+0x20/0x30
Aug 15 00:53:22 lima-murphy-cli-01 kernel: __irq_exit_rcu+0xfc/0x124
Aug 15 00:53:22 lima-murphy-cli-01 kernel: irq_exit_rcu+0x14/0x1c
Aug 15 00:53:22 lima-murphy-cli-01 kernel: el1_interrupt+0x48/0xa4
Aug 15 00:53:22 lima-murphy-cli-01 kernel: el1h_64_irq_handler+0x14/0x20
Aug 15 00:53:22 lima-murphy-cli-01 kernel: el1h_64_irq+0x78/0x7c
Aug 15 00:53:22 lima-murphy-cli-01 kernel: cpuidle_idle_call+0xb4/0x1c0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: do_idle+0xb4/0x120
Aug 15 00:53:22 lima-murphy-cli-01 kernel: cpu_startup_entry+0x28/0x30
Aug 15 00:53:22 lima-murphy-cli-01 kernel: secondary_start_kernel+0x148/0x150
Aug 15 00:53:22 lima-murphy-cli-01 kernel: __secondary_switched+0x6c/0x70
Aug 15 00:53:22 lima-murphy-cli-01 kernel: ---[ end trace 7ce1380af930c7d1 ]---
Aug 15 00:53:22 lima-murphy-cli-01 kernel: virtio_net virtio2 lima0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 6080000 usecs ago
Aug 15 00:53:27 lima-murphy-cli-01 kernel: virtio_net virtio2 lima0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 11040000 usecs ago
```
The below message repeats for ever.
```
Aug 15 00:53:27 lima-murphy-cli-01 kernel: virtio_net virtio2 lima0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, xxxxxxxx usecs ago
```
```
[ranjan@lima-murphy-cli-01 ~]$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.4 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.4 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
```
```
$ limactl --version
limactl version 0.22.0
$ brew info socket_vmnet lima| head -n 1
==> socket_vmnet: stable 1.1.4 (bottled), HEAD [keg-only]
$ sw_vers
ProductName: macOS
ProductVersion: 14.5
BuildVersion: 23F79
$ sysctl -n machdep.cpu.brand_string
Apple M3 Pro
```
Contributor guide
Research direction
Start by reproducing cross-VM traffic stalls with Lima shared networking through socket_vmnet on the reported macOS, Lima, and Rocky Linux versions. Use the provided kernel and virtio_net timeout logs to investigate; done means cross-VM connectivity remains usable without repeated TX timeout messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, macos
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100