Receive side scaling possibly breaking RFC 8305 ("Happy Eyeballs")

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guidelines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

When using Receive Side Scaling - there's a possible issue with RFC 8305 ("Happy Eyeballs").
Specifically, it is a technique for reducing user-visible delays on dual-stack hosts.

What this does in a nutshell, is to fire off a DNS query to OPNsense, e.g.:

15:37:36.222587 IP 192.168.100.72.54242 > 192.168.100.254.53: 21565+ A? something.tld.be. (36)
15:37:36.222593 IP 192.168.100.72.54242 > 192.168.100.254.53: 52287+ AAAA? something.tld.be. (36)

Notice the same source port, and requesting the A and AAAA record at the same time.

When you have RSS enabled, the following answer comes back:

15:37:36.222897 IP 192.168.100.254.53 > 192.168.100.72.54242: 52287* 0/0/0 (36)
15:37:41.227280 IP 192.168.100.72.54242 > 192.168.100.254.53: 21565+ A? something.tld.be. (36)
15:37:41.227960 IP 192.168.100.254.53 > 192.168.100.72.54242: 21565* 1/0/0 A 192.168.100.254 (52)

Note that for the AAAA one, we immediately get NXDOMAIN, but no reply for the A-record, so after 5 seconds, the client asks for the A record again, to which it immediately gets a response. The user experiences this as a delay in initial connection to a resource (site / ssh / ... doesn't matter).

When you do the same test with RSS disabled:

15:53:29.997871 IP 192.168.100.72.49127 > 192.168.100.254.53: 26414+ A? something.tld.be. (36)
15:53:29.997883 IP 192.168.100.72.49127 > 192.168.100.254.53: 33580+ AAAA? something.tld.be. (36)
15:53:29.998256 IP 192.168.100.254.53 > 192.168.100.72.49127: 26414* 1/0/0 A 192.168.100.254 (52)
15:53:29.998257 IP 192.168.100.254.53 > 192.168.100.72.49127: 33580* 0/0/0 (36)

Here, you properly get the NXDOMAIN for the AAAA record, and a response for the A record, within a handful of milliseconds.

To Reproduce

Steps to reproduce the behavior:

1. Enable RSS in a dual-stack network
2. Use a recent linux device
3. issue a simple "ping localdevice"

Expected behavior

RSS scaling should not have issues with Happy Eyeballs

Describe alternatives you considered

Disabling RSS fixes the issue, although preferably, this should get fixed.

Screenshots

n/a

Relevant log files

n/a

Additional context

Using IXL-driver on intel X710

Environment

Software version used and hardware type if relevant, e.g.:

26.1.8_5 (amd64)
Intel X710 with Link Aggragation and CARP.

Start by reproducing the dual-stack DNS test with RSS enabled and disabled on the Intel X710 using packet captures, then trace the receive-side scaling and IXL driver path responsible for handling the responses. Done means concurrent A and AAAA queries both receive prompt, correct replies with RSS enabled, without requiring a retry.