opnsense / opnsense/src

Firewall stops forwarding traffic and web GUI becomes unreachable after upgrade to 26.7

Open
#326 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
505
Forks
202
PR merge metrics
No merged PRs in 30d

Description

Firewall stops forwarding traffic and web GUI becomes unreachable after upgrade to 26.7.x - no NAT/port-forwarding configured

The non-AI part...

Hi. I'm the human behind this. I used AI to help do some troubleshooting on this bug. I have an Opnsense router hosted in a Proxmox environment that stops forwarding traffic after upgrading to any version of 26.7. Everything is fine with the 26.1 chain of code. After upgrading, SSH continues to work, but the internal http servers aren't accessible, and the device doesn't forward traffic. I've included as much detail as I can below. If you have any questions, please let me know. This device isn't used after hours, so I can reproduce the issue and capture more information if you want it.

Per the notes on filing a bug report, I have...

  • Read the guidelines for contributing to Github
  • I don't see any duplicate issues in the list of open or closed issues
  • I used Claude Sonet 5 to create some of the text. I have proof read all text, and entered the commands in the router myself. I'm an experienced network engineer, but I don't have a ton of experience with Opnsense.

Summary

After upgrading from a working 26.1.11_10 install to 26.7.1_1, the firewall stops forwarding traffic and the admin web GUI becomes unreachable after running after 5 - 15 minutes. SSH access continues to work throughout - the host itself does not freeze or become unresponsive, only traffic forwarding and the web GUI are affected. Power-cycling the VM does not reliably restore service. Rolling back to 26.1.11_10 resolved the issue.

This configuration has no NAT/rdr rules and no port-forwarded servers of any kind, and Suricata's capture mode is confirmed set to netmap (IPS), not Divert - so there is no divert-to rule in the ruleset at all either. This rules out the specific mechanism described in opnsense/src#303 (rdr + stateful divert-to causing a state-insert block), which core#10615 was closed in favor of, on both counts. Since the symptoms otherwise match #10615/#303 closely (traffic silently stops passing despite matching rules, GUI/web services become unreachable, pf state table is never exhausted), but neither of #303's reproduction ingredients (rdr, divert-to) are present here, this looks like a related-but-distinct trigger path in the same FreeBSD 15.1 pf regression family. Two candidate differentiators in this configuration: the netmap-based Suricata IPS integration (rather than the divert-socket path #303 covers), and/or Traffic Shaper (Limiters/dummynet queues), which is actively configured on this firewall and - like rdr/divert-to - involves pf rule actions (dnqueue/dnpipe) that participate in state creation. dmesg showed repeated config_aqm Unable to configure flowset, flowset busy! messages at boot, consistent with some kind of instability in the dummynet/AQM flowset configuration path on this FreeBSD 15.1 base.

Environment

  • OPNsense: upgraded 26.1.11_10 → 26.7.1_1 (issue appears both on 26.7.1 and 26.7.3; confirmed absent on 26.1.11_10 both before the upgrade and after rolling back)
  • Platform: VM under Proxmox/KVM, virtio NICs (vtnet0/vtnet1 in FreeBSD)
  • WAN and LAN are both routed public/statically-addressed segments; no NAT, no rdr/port-forward rules, no NAT reflection
  • Suricata is installed with capture mode explicitly configured as netmap (IPS) (not Divert) - confirmed via the GUI setting, and observed running under --netmap on the WAN interface at one point. It was also observed not running at all at another point during troubleshooting (crash or failed restart - cause not determined). Since capture mode is netmap, not Divert, there is no divert-to pf rule in this ruleset at all.
  • Traffic Shaper (Limiters) is configured and in active use on this firewall — another pf feature (dnqueue/dnpipe rule actions) that participates in state creation, in the same general family as rdr/divert-to
  • Proxmox's Firewall feature is disabled at the datacenter level.

Steps to reproduce

  1. Upgrade an Opnsense 26.1.11_10 VM to 26.7.1_1 as a Proxmox/KVM VM with virtio NICs, no NAT/port-forward rules configured, Suricata installed.
  2. Leave it running under normal traffic.
  3. After a few minutes traffic forwarding stops and the web GUI becomes unreachable.

(Exact trigger conditions not yet isolated - filing in case the symptom, environment, and ruled-out list help correlate with other reports.)

Expected behavior

Firewall continues forwarding traffic and the GUI remains reachable indefinitely, as it did under 26.1.11_10.

Actual behavior

  • Forwarding stops (this was the primary reported symptom - traffic no longer passes as expected)
  • Admin web GUI becomes unreachable
  • SSH access is unaffected throughout - the OS itself remains responsive
  • A full VM power-cycle did not restore service
  • Rolling back to 26.1.11_10 resolved the issue

Diagnostics collected (values shown are representative; exact IPs redacted/genericized)

pfctl -s info — state table nowhere near exhaustion, state-limit/src-limit counters both 0:

State Table                          Total             Rate
  current entries                     1987
Counters
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s

netstat -i — no interface errors/drops on either NIC:

Name       Mtu Network           Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
vtnet0    1500 <Link#1>          xx:xx:xx:xx:xx:xx    27036     0     0    24948     0     0
vtnet1    1500 <Link#2>          xx:xx:xx:xx:xx:xx   352852     0     0   249530     0     0

top -aSH at time of one check — CPU 99.4% idle, load average 0.18; Suricata threads each under 1% CPU:

CPU:  0.4% user,  0.0% nice,  0.1% system,  0.1% interrupt, 99.4% idle
load averages:    0.18,    0.19,    0.16

ps aux | grep suricata at another point during the incident window — no Suricata process running at all (unclear if this reflects a crash, a failed restart, or unrelated timing):

root    40943   0.0  0.0  14088   2616  0  S+   14:43    0:00.00 grep suricata

dmesg — no panic or watchdog timeout messages found across multiple observed reboots that day, despite repeated full driver re-attach sequences for both vtnet0/vtnet1 indicating the VM had rebooted several times. Also seen at every boot, immediately after ipfw2 initialization and around the same time the interfaces come up:

config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!

This recurred at multiple points in the same boot sequence (not just once), suggesting the Traffic Shaper/dummynet flowset configuration is being applied more than once, or racing with something else, during startup.

curl -v -m 5 http://127.0.0.1/ (checked during a presumed-healthy window, not confirmed during an active failure) returned a normal 301 Moved Permanently → https://127.0.0.1/, and the php-cgi worker pool (~24 idle workers) was present and healthy — so the GUI backend itself was not obviously crashed in that particular check.

What's been ruled out

  • DDoS/flood: no evidence at any capture point - state table far below any limit, no interface drops, no CPU pressure, no single dominant source IP among current states.
  • GUI backend (php-cgi) crash: process pool present and responsive via local curl test, at least during the checks performed.
  • NAT/rdr + divert-to interaction (opnsense/src#303): does not apply - this configuration has no NAT of any kind.
  • Kernel panic / driver watchdog timeout: no matching dmesg entries found across several observed reboots.

Related issues

  • opnsense/core#10615 (closed in favor of opnsense/src#303) - very similar symptom (state-insert,block dropping otherwise-permitted traffic, web services becoming unreachable, state table not exhausted), but that issue's root cause (#303) specifically requires rdr + divert-to, which this configuration doesn't have.
  • opnsense/core#10097 - Suricata Divert mode config/netmap generation inconsistency; likely not directly applicable here since this configuration deliberately uses netmap (IPS) mode rather than Divert, but noted for completeness given the overlap in subsystem.

Fix

Rolling back to 26.1.11_10 resolved the issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the upgrade from OPNsense 26.1.11_10 to 26.7.x in the described Proxmox/KVM setup, then inspect the pf, netmap, Traffic Shaper/dummynet, and AQM behavior. Check dmesg for the reported “config_aqm Unable to configure flowset, flowset busy!” messages and compare forwarding and GUI reachability before and after failure. Done means identifying the trigger and restoring indefinite forwarding and GUI access without a rollback.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
networking, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.