DHCP from illumos zone not working

EDIT: This is blocked on illumos#11990

I tested both DHCP and ICMP on a very hacked up Omicron + OPTE environment. However, after the USDT work and some rust toolchain updates I was no longer able to build that environment. Rather than spend an inordinate amount of time tracking that down I went ahead and pushed anyways. However, I also made a few adjustments to the DHCP/ICMP rule predicates, to make sure they more fully qualify the requests being made by the guest. But since I could no longer stand up my Omicron+OPTE environment I couldn't test it.

Either those additional qualifications were too stringent and broke DHCP or the illumos netstack is doing something slightly different than Linux and I need to account for it. Unfortunately, I also couldn't dump the dhcp layer rules because of #68.

The excerpts below show the zone failing to get a DHCP reply because the xde0 device is failing to match the packet against the dhcp4 layer. This causes it to make it to the router layer which rejects it, as there is no route to the virtual gateway.

This last point has me thinking: we should have a least priority rule (aka high value) in the dhcp4 layer (and really dhcp4 + icmp + arp should probably all be merged into a gateway layer) that predicates on only the destination address of the virtual gateway and performs Drop. That would prevent traffic destined for the virtual gateway from leaking past those first layers. Otherwise, I imagine the router could end up sending it out default route aka the "Internet" Gateway. Also, it just kind of makes sense to constrain the traffic from proceeding any further than it should.

root@iz1:~# ipadm create-addr -t -T dhcp vnic0/v4
ipadm: warning: Communication with dhcpagent timed out

root@sled1:~/dtrace# ./opte-trace opte-port-process.d 
DIR NAME         FLOW                                        MBLK               RESULT
Out xde0         UDP,0.0.0.0:68,255.255.255.255:67           0xfffffe03964b2540 Drop { reason: Layer { name: "router" } }
Out xde0         UDP,0.0.0.0:68,255.255.255.255:67           0xfffffe03964b2540 Drop { reason: Layer { name: "router" } }

root@sled1:~/dtrace# ./opte-trace opte-rule-match.d 
MATCH  DIR LAYER        FLOW                                        ACTION
NO     out dhcp4        UDP,0.0.0.0:68,255.255.255.255:67           --
NO     out icmp         UDP,0.0.0.0:68,255.255.255.255:67           --
NO     out arp          UDP,0.0.0.0:68,255.255.255.255:67           --
YES    out router       UDP,0.0.0.0:68,255.255.255.255:67           DENY