TCP: Total bytes sent (n_send) and received (n_recv) always mis-matched even no dropped/error pkts in network stacks between two end-points.

For TCP protocol test, the total bytes sent (n_send) and received (n_recv) reported at the end of the test are not always matched.
(Note that UDP also have the same issue, although UDP is unreliable protocol, so it's subjected to pkt loss).

I tested TCP transfer from one VM to another remote VM, with 1 tcp stream (1 sender thread/connection -> 1 receiver/connection) with sender and receiver buffer size both set to 1B (-b 1), and ran for 60 second. Below shows the cmdlines setup:

ntttcp -s -m 1,,10.231.111.24 -V --show-tcp-retrans --show-nic-packets eth1 -p 5021 -b 1 -n 1 -t 60
ntttcp -r -m 1,,0.231.111.24 -V --show-tcp-retrans --show-nic-packets eth0 -p 5021 -b 1

Sender report:
05:03:54 INFO: test duration :60.00 seconds
05:03:54 INFO: total bytes :39287223
05:03:54 INFO: throughput :5.24Mbps

Receiver report:
05:03:51 INFO: test duration :60.00 seconds
05:03:51 INFO: total bytes :38750015
05:03:51 INFO: throughput :5.17Mbps

Sender always report higher total bytes than receiver.
I ran multiple / several times, and they all showed mis-matches between the sender and receiver.

During the test run, I collected the tcpdump traces on both sender and receiver VMs, as well as monitoring the network interface statistics (e.g. ip -s -d link show eth0). They did not report any dropped or error packets.

I also collected pktmon traces on both sender and receiver hosts, but the traces did not indicate any issues like dropped or error.

Attached the test run logs, tcpdump traces, interface statistics (host sides traces are too big):

Tcp_1stream_TotalBytesSendAndRecvdMismatch_1ByteBufferSize.zip

Note that below is one issue that can contribute to the mis-match, but is not the main cause of the above mis-match. I added additional logging in the error return case to output any pkts received prior to the error return. And the mis-match existed even without hitting the error case statement.

One Issue noticed:
It looks like in the n_send and n_recv functions in tcpstream.c, there can be bytes missing from the calculation. For example, in below n_recv function in the 1st else statement which it returns just the error code, but the error could happen after some packets have been received during the while loop, thus the number of those received packets were not returned. Similar symptom on the n_send function as well.

int n_recv(int fd, char *buffer, size_t total)
{
register ssize_t rtn;
register size_t left = total;

while (left > 0) {
	rtn = recv(fd, buffer, left, 0);
	if (rtn < 0) {
		if (errno == EINTR || errno == EAGAIN) {
			break;
		} else {
			printf("socket read error: %d\n", errno);
			return ERROR_NETWORK_READ;
		}
	} else if (rtn == 0)
		break;

	left -= rtn;
	buffer += rtn;
}

return total - left;

}

Start in tcpstream.c and inspect the n_send and n_recv accounting paths, including their error and partial-transfer returns. Reproduce the one-stream TCP test with 1-byte buffers using the command lines in the report, then compare the sender and receiver totals with the attached tcpdump and interface traces. Done means the reported totals account for all transferred bytes and the mismatch is explained or corrected.