opnsense / opnsense/src

panic: page fault in sysctl_viflist() when reading net.inet.ip.viftable ("netstat -g") with igmpproxy running

Open
#322 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
505
Forks
202
PR merge metrics
No merged PRs in 30d

Description

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

Describe the bug

Reading the multicast virtual interface list via sysctl panics the kernel with a page
fault while igmpproxy is running. The faulting call chain is __sysctlbyname ->
sysctl_viflist() -> __rw_rlock_hard(), i.e. the fault happens while the sysctl
handler takes a read lock, before any data is returned to userland.

The trigger in my case was netstat -g, which is an ordinary read-only diagnostic
command. An unprivileged-looking, read-only operation should not be able to take the
router down.

Sequence that produced it:

  1. IPTV/multicast setup: igmpproxy with re1 upstream (WAN, DHCP) and re2
    downstream (LAN segment carrying the IPTV STB), quickleave enabled.
  2. re2 had just gained carrier a few minutes earlier (cable newly connected).
  3. configctl igmpproxy start — igmpproxy started successfully and was confirmed
    running.
  4. Roughly 5 seconds later, netstat -g -f inet | head -15 was executed over SSH.
  5. Immediate kernel panic, box rebooted.

Last known working version: unknown. I have not previously run netstat -g on this
system, so I cannot say whether an earlier release behaved differently.

To Reproduce

Steps to reproduce the behavior:

  1. Configure igmpproxy with an upstream and a downstream phyint (Services -> IGMP Proxy).
  2. Start the service and confirm it is running.
  3. From a shell, run netstat -g (equivalently, read the net.inet.ip.viftable sysctl).
  4. Kernel panics with Fatal trap 12: page fault while in kernel mode.

I have only observed this once and have not attempted to reproduce it, because
doing so takes the router — and the whole network behind it — offline. I am happy to
reproduce it in a maintenance window if that would help, or to test a patch.

Likely minimal reproducer for someone with a lab box:

sysctl net.inet.ip.viftable

Expected behavior

netstat -g / reading net.inet.ip.viftable should return the multicast VIF table (or
an empty result), never panic the kernel.

Describe alternatives you considered

The only workaround I have is to avoid netstat -g and netstat -M entirely on this
system. That is acceptable for me because nothing in OPNsense itself appears to call
netstat -g — I grepped /usr/local/opnsense/ and /usr/local/www/ and found no
references — so the web GUI does not appear to be able to trigger this. It is only
reachable by an administrator running the command by hand.

Screenshots

N/A — console output and textdump are below/attached.

Relevant log files

Backtrace from /var/crash/textdump.tar.1 (ddb.txt):

db:0:kdb.enter.default>  bt
Tracing pid 48533 tid 105119 td 0xfffff80157b59780
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00da9898d0
panic() at panic+0x43/frame 0xfffffe00da989930
trap_pfault() at trap_pfault+0x37d/frame 0xfffffe00da989980
calltrap() at calltrap+0x8/frame 0xfffffe00da989980
--- trap 0xc, rip = 0xffffffff80c01d74, rsp = 0xfffffe00da989a50, rbp = 0xfffffe00da989ad0 ---
__rw_rlock_hard() at __rw_rlock_hard+0xe4/frame 0xfffffe00da989ad0
sysctl_viflist() at sysctl_viflist+0x55/frame 0xfffffe00da989b00
sysctl_root_handler_locked() at sysctl_root_handler_locked+0x91/frame 0xfffffe00da989b50
sysctl_root() at sysctl_root+0x268/frame 0xfffffe00da989bd0
userland_sysctl() at userland_sysctl+0x1a6/frame 0xfffffe00da989c70
kern___sysctlbyname() at kern___sysctlbyname+0x226/frame 0xfffffe00da989dc0
sys___sysctlbyname() at sys___sysctlbyname+0x2d/frame 0xfffffe00da989e00
amd64_syscall() at amd64_syscall+0x126/frame 0xfffffe00da989f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00da989f30
--- syscall (570, FreeBSD ELF64, __sysctlbyname), rip = 0x227fa15f5baa, rsp = 0x227f9aee4b18, rbp = 0x227f9aee5350 ---

panic.txt:

page fault

version.txt:

FreeBSD 15.1-RELEASE-p3 stable/26.7-n283797-10516aac4165 SMP

show pcpu confirms the faulting thread is the netstat process:

cpuid        = 1
curthread    = 0xfffff80157b59780: pid 48533 tid 105119 critnest 1 "netstat"
curvnet      = 0xfffff800041e5ac0

Full textdump (ddb.txt, msgbuf.txt, panic.txt, version.txt, config.txt) attached.

Additional context

The fault address is reached from sysctl_viflist()+0x55 inside __rw_rlock_hard(),
which suggests the sysctl handler is taking a read lock on an rwlock that is not (yet)
valid for this VNET rather than dereferencing bad table data. I have not verified this
— it is only what the backtrace suggests.

Possibly relevant environment details:

  • The system runs with VIMAGE/VNET (curvnet is non-NULL in the dump).
  • re2, the igmpproxy downstream interface, had only just gained carrier when the
    panic occurred, and igmpproxy had been running for a few seconds. A race between
    VIF creation and the sysctl read is plausible but unproven.
  • IPTV via igmpproxy itself works correctly on this release, both before and after the
    panic — multicast is forwarded from re1 to re2 and the STB plays normally. The
    panic is specific to reading the VIF list.
  • net.inet.igmp.default_version=2 is set as a tunable (needed by my ISP). This is
    unrelated to the panic but is present on the system.

igmpproxy.conf in use at the time:

quickleave
phyint re1 upstream ratelimit 0 threshold 1
altnet 0.0.0.0/1
altnet 128.0.0.0/1
phyint re2 downstream ratelimit 0 threshold 1
altnet 10.1.0.0/16
phyint re3 disabled
phyint re0 disabled
phyint vtnet0 disabled

Packages: igmpproxy-0.4_3,1, os-igmp-proxy-1.5_6.

Environment

OPNsense 26.7.3_8 (amd64)
FreeBSD 15.1-RELEASE-p3 stable/26.7-n283797-10516aac4165 SMP
Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 8 CPUs
Guest: KVM (kern.vm_guest: kvm) under Proxmox VE, VIMAGE/VNET in use
Network: 4x Realtek RTL8111/8168/8211/8411 (re0-re3) via PCI passthrough,
plus 1x VirtIO (vtnet0)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached ddb.txt and the sysctl_viflist() entry point; review the faulting __rw_rlock_hard() call in the context of VIMAGE/VNET and igmpproxy-created VIFs. Reproduce only on a lab system using an igmpproxy upstream/downstream setup and sysctl net.inet.ip.viftable. Done means the sysctl returns a VIF table or empty result without a kernel panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.