esnet / esnet/iperf

slow udp stream generation with message size 1400B

Open
#899 4 comments 3 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.8k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Description

# Context

* Version of iperf3: **master**

* Hardware: 10G ixgbe nic, other parts are probably not relevant

* Operating system (and distribution, if any): RHEL8.0 but should be reproducible on other Linux distros

* Other relevant information (for example, non-default compilers,
libraries, cross-compiling, etc.):

# Bug Report

* Expected Behavior

when running ```iperf3 -c -B -u -b 0 -t 10 -l 1400 -A ```

I'd expect the iperf generator to either reach line rate throughput (close to 10Gbps) or to reach maximum utilization of a single CPU (processor isn't fast enough to generate more packets per second)

* Actual Behavior

Generator only generates 4.3gbps while the utilization of the single cpu core is only ~40%.

Running netperf in a similar configuration - basic ipv4 udp stream with 1400 messages, does reach line rate without issues.

* Steps to Reproduce

1. configure a simple network of 2 hosts with a connection capable of 10Gbps
2. start an iperf3 server
3. run the client with the following command ```iperf3 -c -B -u -b 0 -t 10 -l 1400 -A ``` replacing the <> values with whatever you configured

* Possible Solution

I investigated this issue with a colleague who has experience with udp kernel development and we found out that this seems to happen because of an unfortunate combination of UDP stream generation burstiness and the fact that every 10 test stream ```write()``` calls a ```select()``` gets called to check on the control connection to the server.

What seems to happen is that some amount of writes to the test socket fills out it's buffer for a short time at which point the select call suspends the iperf client process and re-waking the process takes a bit of time, considering that this happens every 10 writes we think this can affect the generator performance.

We made this conclusion after:
* comparing iperf and netperf strace
* both create and configure the socket in comparable ways
* one difference is that iperf uses ```write``` and netperf uses ```sendto``` but this shouldn't have the measured impact
* netperf doesn't interlace ```sendto``` with ```select``` calls, which means that when a socket buffer fills up for a moment the sendto call blocks and we suspect that re-waking from this kind of blocking might be faster than from the select call in iperf
* we found that the **10** value comes from https://github.com/esnet/iperf/blob/master/src/iperf_api.c#L2331 and when changed to a larger value (e.g. 1000) the issue dissappears

When looking for possible solutions I found that using a larger send buffer size (using the -w argument) can also work around the issue, however considering that netperf doesn't configure this and uses the default send buffer size I don't consider this a valid solution.

Configuring a burst packet value using ```-b 0/1000``` can override the multisend variable, however this currently doesn't work and I submitted a pull request #898 to fix this. However, I'm not sure if this is a good and *intended* way to configure iperf to be faster for UDP streams, maybe the "multisend" variable should also be configurable via a separate CLI argument?

I'm willing to look into implementing a solution and sending a pull request after discussing it here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.