iperf3 Bandwidth fluctuation with mutiple clients (25G NIC)
- Dominant language
- C
- Stars
- 8.8k
- Forks
- 1.5k
- PR merge metrics
- No merged PRs in 30d
Description
#Context
The server creates two iperf3 servers. When bound to the same cpu, the bandwidth is stable, but when bound to different cpus, bandwidth fluctuations will occur
* Version of iperf3:
```
[root@node-3 queues]# iperf3 -v
iperf 3.1.7
Linux node-3.domain.tld 4.18.0-147.5.1.el8_1.5es.4.aarch64 #1 SMP Tue Jun 2 09:55:33 UTC 2020 aarch64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing
```
* Hardware:
```
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing
[root@node-3 queues]# lscpu
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 1
Core(s) per socket: 32
Socket(s): 2
NUMA node(s): 4
CPU max MHz: 2600.0000
CPU min MHz: 200.0000
L1d cache: 64K
L1i cache: 64K
L2 cache: 512K
L3 cache: 32768K
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
NUMA node2 CPU(s): 32-47
NUMA node3 CPU(s): 48-63
```
* Operating system (and distribution, if any):
```
[root@node-3 queues]# uname -r
4.18.0-147.5.1.el8_1..aarch64
```
### Steps to Reproduce(Take my environment as an example)
**Have been manually bound to irq affinity**
### case 1 in different cpu:
**server:(node-3)**
```
nohup iperf3 -A 14/14 -p 5201 &
nohup iperf3 -A 15/15 -p 5202 &
```
**client(node-1,node-2)**
```
node-1:iperf3 -c 192.168.236.4 -i 1 -t 10000 -b 10G -P 1 -p 5201
node-2:iperf3 -c 192.168.236.4 -i 1 -t 10000 -b 10G -P 1 -p 5202
```
**Test Results**
* node-1:

* node-2

### case 2 same cpu:
**server:(node-3)**
```
nohup iperf3 -A 14/14 -p 5201 &
nohup iperf3 -A 14/14 -p 5202 &
```
**client(node-1,node-2)**
```
node-1:iperf3 -c 192.168.236.4 -i 1 -t 10000 -b 10G -P 1 -p 5201
node-2:iperf3 -c 192.168.236.4 -i 1 -t 10000 -b 10G -P 1 -p 5202
```
**Test Results**
* node-1:

* node-2:

Contributor guide
Assessment
This issue has not been assessed yet.