weird behavior on 1500 MTU underlay with uft-hit probe enabled

NOTE: This issue does not affect opte on 9000 MTU links, I stumbled onto this when I accidentally ran across 1500 MTU links. I'm filing this issue as something that we might want to follow up on just to understand this odd behavior and make sure it's not a sign of something more severe (like miscompilation).

Recently I setup a helios/propolis-standalone/opte bench rig in my homelab. During initial setup I ran into a connectivity issue that had me stumped. When running iperf between two guests (on separate hosts) I found that each time the stream would initially send some data but then quickly drop to 0 bits-per-second.

tseng:~# iperf3 -c 10.0.1.113 -B 10.0.1.112 -t10 -P1
Connecting to host 10.0.1.113, port 5201
[  5] local 10.0.1.112 port 46079 connected to 10.0.1.113 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  62.9 MBytes   527 Mbits/sec   21   1.41 KBytes
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  62.9 MBytes  52.7 Mbits/sec   24            sender
[  5]   0.00-10.01  sec  59.5 MBytes  49.9 Mbits/sec                  receiver

Perplexed, I started running the various opte dtrace scripts included in this repo in an attempt to track down the issue. Then, when I ran uft-hit.d on the receiving host, the traffic suddenly started flowing! I was able to replicate this on multiple runs, below is an example where I started the benchmark with the probe enabled, and then disabled it around the halfway mark.

tseng:~# iperf3 -c 10.0.1.113 -B 10.0.1.112 -t10 -P1
Connecting to host 10.0.1.113, port 5201
[  5] local 10.0.1.112 port 44227 connected to 10.0.1.113 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   256 MBytes  2.15 Gbits/sec    0   2.40 MBytes
[  5]   1.00-2.00   sec   275 MBytes  2.31 Gbits/sec    0   2.54 MBytes
[  5]   2.00-3.00   sec   275 MBytes  2.30 Gbits/sec    0   2.54 MBytes
[  5]   3.00-4.00   sec   275 MBytes  2.31 Gbits/sec    0   2.67 MBytes
[  5]   4.00-5.00   sec   277 MBytes  2.33 Gbits/sec    0   2.67 MBytes
[  5]   5.00-6.00   sec   277 MBytes  2.32 Gbits/sec    0   2.67 MBytes
[  5]   6.00-7.00   sec  75.5 MBytes   633 Mbits/sec    4   1.41 KBytes
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.67 GBytes  1.44 Gbits/sec    6            sender
[  5]   0.00-10.00  sec  1.67 GBytes  1.43 Gbits/sec                  receiver

I can think of no reason why enabling this probe should change anything about the datapath: except for possible miscompilation?

Anyways, I eventually got a packet capture of a run (without the probe enabled) and found my initial problem: the connection establishes fine and initially sends some LSO-sized packets just fine, but then the client guest goes to send a packet of length 1448 and no forward progress is made after that. This was because my cxgbe links were set to a 1500 MTU, and when the guest goes to send the 1448 packet opte encaps it resulting in a packet of length 1592 -- to large to traverse the link. At this point the guest client cannot make any progress because it tries to retransmit the same broken packet, and then eventually the iperf test ends.

But the real question is: why would enabling the uft-hit probe on the server change this behavior in any way? As I'm writing up this issue now I realized I didn't grab a capture on the receivers side, and I'm really not sure how doing anything on the receiver could unblock the stream when it's the client trying to send a too large frame across the wire. It really makes no sense, but I don't have the time to dig into this right now.

Start by reproducing the iperf3 failure with a 1500 MTU underlay and the uft-hit.d probe enabled and disabled. Capture traffic on both sides and trace the probe's effect on the datapath; done means the probe-related behavior is explained and any underlying defect is corrected.