esnet / esnet/iperf

iperf3 Bandwidth fluctuation with mutiple clients (25G NIC)

Open
#1,047 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.8k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Description

#Context
The server creates two iperf3 servers. When bound to the same cpu, the bandwidth is stable, but when bound to different cpus, bandwidth fluctuations will occur
* Version of iperf3:
```
[root@node-3 queues]# iperf3 -v
iperf 3.1.7
Linux node-3.domain.tld 4.18.0-147.5.1.el8_1.5es.4.aarch64 #1 SMP Tue Jun 2 09:55:33 UTC 2020 aarch64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing
```
* Hardware:
```
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing
[root@node-3 queues]# lscpu
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 1
Core(s) per socket: 32
Socket(s): 2
NUMA node(s): 4
CPU max MHz: 2600.0000
CPU min MHz: 200.0000
L1d cache: 64K
L1i cache: 64K
L2 cache: 512K
L3 cache: 32768K
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
NUMA node2 CPU(s): 32-47
NUMA node3 CPU(s): 48-63
```
* Operating system (and distribution, if any):
```
[root@node-3 queues]# uname -r
4.18.0-147.5.1.el8_1..aarch64
```
### Steps to Reproduce(Take my environment as an example)
**Have been manually bound to irq affinity**
### case 1 in different cpu:
**server:(node-3)**
```
nohup iperf3 -A 14/14 -p 5201 &
nohup iperf3 -A 15/15 -p 5202 &
```
**client(node-1,node-2)**
```
node-1:iperf3 -c 192.168.236.4 -i 1 -t 10000 -b 10G -P 1 -p 5201
node-2:iperf3 -c 192.168.236.4 -i 1 -t 10000 -b 10G -P 1 -p 5202
```
**Test Results**
* node-1:
![image](https://user-images.githubusercontent.com/10482973/91658645-91fa9c80-eafc-11ea-8e8d-c9424ca777d0.png)

* node-2
![image](https://user-images.githubusercontent.com/10482973/91658680-f0c01600-eafc-11ea-9970-66f145b61f39.png)

### case 2 same cpu:
**server:(node-3)**
```
nohup iperf3 -A 14/14 -p 5201 &
nohup iperf3 -A 14/14 -p 5202 &
```
**client(node-1,node-2)**
```
node-1:iperf3 -c 192.168.236.4 -i 1 -t 10000 -b 10G -P 1 -p 5201
node-2:iperf3 -c 192.168.236.4 -i 1 -t 10000 -b 10G -P 1 -p 5202
```
**Test Results**
* node-1:
![image](https://user-images.githubusercontent.com/10482973/91658686-09303080-eafd-11ea-95c6-1efc4051788a.png)

* node-2:
![image](https://user-images.githubusercontent.com/10482973/91658694-10efd500-eafd-11ea-8cae-5e7e16056b16.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.