panic: page fault in sysctl_viflist() when reading net.inet.ip.viftable ("netstat -g") with igmpproxy running
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 505
- Forks
- 202
- PR merge metrics
- No merged PRs in 30d
Description
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- I have read the contributing guide lines at https://github.com/opnsense/src/blob/master/CONTRIBUTING.md
- I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/src/issues?q=is%3Aissue
Describe the bug
Reading the multicast virtual interface list via sysctl panics the kernel with a page
fault while igmpproxy is running. The faulting call chain is __sysctlbyname ->
sysctl_viflist() -> __rw_rlock_hard(), i.e. the fault happens while the sysctl
handler takes a read lock, before any data is returned to userland.
The trigger in my case was netstat -g, which is an ordinary read-only diagnostic
command. An unprivileged-looking, read-only operation should not be able to take the
router down.
Sequence that produced it:
- IPTV/multicast setup:
igmpproxywithre1upstream (WAN, DHCP) andre2
downstream (LAN segment carrying the IPTV STB),quickleaveenabled. re2had just gained carrier a few minutes earlier (cable newly connected).configctl igmpproxy start— igmpproxy started successfully and was confirmed
running.- Roughly 5 seconds later,
netstat -g -f inet | head -15was executed over SSH. - Immediate kernel panic, box rebooted.
Last known working version: unknown. I have not previously run netstat -g on this
system, so I cannot say whether an earlier release behaved differently.
To Reproduce
Steps to reproduce the behavior:
- Configure
igmpproxywith an upstream and a downstreamphyint(Services -> IGMP Proxy). - Start the service and confirm it is running.
- From a shell, run
netstat -g(equivalently, read thenet.inet.ip.viftablesysctl). - Kernel panics with
Fatal trap 12: page fault while in kernel mode.
I have only observed this once and have not attempted to reproduce it, because
doing so takes the router — and the whole network behind it — offline. I am happy to
reproduce it in a maintenance window if that would help, or to test a patch.
Likely minimal reproducer for someone with a lab box:
sysctl net.inet.ip.viftable
Expected behavior
netstat -g / reading net.inet.ip.viftable should return the multicast VIF table (or
an empty result), never panic the kernel.
Describe alternatives you considered
The only workaround I have is to avoid netstat -g and netstat -M entirely on this
system. That is acceptable for me because nothing in OPNsense itself appears to call
netstat -g — I grepped /usr/local/opnsense/ and /usr/local/www/ and found no
references — so the web GUI does not appear to be able to trigger this. It is only
reachable by an administrator running the command by hand.
Screenshots
N/A — console output and textdump are below/attached.
Relevant log files
Backtrace from /var/crash/textdump.tar.1 (ddb.txt):
db:0:kdb.enter.default> bt
Tracing pid 48533 tid 105119 td 0xfffff80157b59780
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00da9898d0
panic() at panic+0x43/frame 0xfffffe00da989930
trap_pfault() at trap_pfault+0x37d/frame 0xfffffe00da989980
calltrap() at calltrap+0x8/frame 0xfffffe00da989980
--- trap 0xc, rip = 0xffffffff80c01d74, rsp = 0xfffffe00da989a50, rbp = 0xfffffe00da989ad0 ---
__rw_rlock_hard() at __rw_rlock_hard+0xe4/frame 0xfffffe00da989ad0
sysctl_viflist() at sysctl_viflist+0x55/frame 0xfffffe00da989b00
sysctl_root_handler_locked() at sysctl_root_handler_locked+0x91/frame 0xfffffe00da989b50
sysctl_root() at sysctl_root+0x268/frame 0xfffffe00da989bd0
userland_sysctl() at userland_sysctl+0x1a6/frame 0xfffffe00da989c70
kern___sysctlbyname() at kern___sysctlbyname+0x226/frame 0xfffffe00da989dc0
sys___sysctlbyname() at sys___sysctlbyname+0x2d/frame 0xfffffe00da989e00
amd64_syscall() at amd64_syscall+0x126/frame 0xfffffe00da989f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00da989f30
--- syscall (570, FreeBSD ELF64, __sysctlbyname), rip = 0x227fa15f5baa, rsp = 0x227f9aee4b18, rbp = 0x227f9aee5350 ---
panic.txt:
page fault
version.txt:
FreeBSD 15.1-RELEASE-p3 stable/26.7-n283797-10516aac4165 SMP
show pcpu confirms the faulting thread is the netstat process:
cpuid = 1
curthread = 0xfffff80157b59780: pid 48533 tid 105119 critnest 1 "netstat"
curvnet = 0xfffff800041e5ac0
Full textdump (ddb.txt, msgbuf.txt, panic.txt, version.txt, config.txt) attached.
Additional context
The fault address is reached from sysctl_viflist()+0x55 inside __rw_rlock_hard(),
which suggests the sysctl handler is taking a read lock on an rwlock that is not (yet)
valid for this VNET rather than dereferencing bad table data. I have not verified this
— it is only what the backtrace suggests.
Possibly relevant environment details:
- The system runs with VIMAGE/VNET (
curvnetis non-NULL in the dump). re2, the igmpproxy downstream interface, had only just gained carrier when the
panic occurred, and igmpproxy had been running for a few seconds. A race between
VIF creation and the sysctl read is plausible but unproven.- IPTV via igmpproxy itself works correctly on this release, both before and after the
panic — multicast is forwarded fromre1tore2and the STB plays normally. The
panic is specific to reading the VIF list. net.inet.igmp.default_version=2is set as a tunable (needed by my ISP). This is
unrelated to the panic but is present on the system.
igmpproxy.conf in use at the time:
quickleave
phyint re1 upstream ratelimit 0 threshold 1
altnet 0.0.0.0/1
altnet 128.0.0.0/1
phyint re2 downstream ratelimit 0 threshold 1
altnet 10.1.0.0/16
phyint re3 disabled
phyint re0 disabled
phyint vtnet0 disabled
Packages: igmpproxy-0.4_3,1, os-igmp-proxy-1.5_6.
Environment
OPNsense 26.7.3_8 (amd64)
FreeBSD 15.1-RELEASE-p3 stable/26.7-n283797-10516aac4165 SMP
Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 8 CPUs
Guest: KVM (kern.vm_guest: kvm) under Proxmox VE, VIMAGE/VNET in use
Network: 4x Realtek RTL8111/8168/8211/8411 (re0-re3) via PCI passthrough,
plus 1x VirtIO (vtnet0)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attached ddb.txt and the sysctl_viflist() entry point; review the faulting __rw_rlock_hard() call in the context of VIMAGE/VNET and igmpproxy-created VIFs. Reproduce only on a lab system using an igmpproxy upstream/downstream setup and sysctl net.inet.ip.viftable. Done means the sysctl returns a VIF table or empty result without a kernel panic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100