esnet / esnet/iperf

bandwidth limit overshoot after micro outages

Open
#1,747 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.8k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Description

# Context

* Version of iperf3: 3.9 (rhel), 3.14 (ubuntu), 3.8.1 (darwin)
* Hardware: various
* Operating system (and distribution, if any): various
* Other relevant information: from package managers

# Bug Report

The --bitrate option is misleading, or not documented well, depends how you want to classify it. Instead of being maximum target bitrate, it is a long-term averaging target - it can overdrive without a limit until the long-term average has settled. The comment on optional burst rate ("can temporarily exceed the specified bandwidth limit") implies the non-burst version does not temporarily exceed the intended bandwidth.

## Actual Behavior

If you have small outages on a network (eg 10 seconds), the bitrate throttle will attempt to catch up on the lost traffic by behaving if no throttle limit exists, driving the traffic as fast as it possibly can until the _average_ bitrate since the start of the matches the long-term target bitrate. This seems to make sense looking at iperf_check_throttle, which calculates the average since the start time.

This doesn't look too exciting on a LAN or high-speed network (maybe a second or so at maximum), but on a slower WAN it may saturate the link for many minutes trying make up for the lost data.

On a LAN, the overshoot looks like a quantisation error - just filling up the congestion window for a short blip.

Unfortunately I only have an example on a LAN for the moment. I can generate a WAN looking example if required.

```bash
❯ iperf3-darwin -c beep --bitrate 20M --time 500
Connecting to host beep, port 5201
[ 7] local 2403:5801:xxx:x:xxxx:xxxx:xxxx:xxxx port 63316 connected to 2403:5801:xxx:x::x port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd RTT
[ 7] 0.00-1.00 sec 2.50 MBytes 21.0 Mbits/sec 2 1.80 MBytes 4ms
[ 7] 1.00-2.00 sec 2.38 MBytes 19.9 Mbits/sec 3 3.94 MBytes 6ms
[ 7] 2.00-3.00 sec 2.38 MBytes 19.9 Mbits/sec 2 5.90 MBytes 10ms
[ 7] 3.00-4.00 sec 2.38 MBytes 19.9 Mbits/sec 2 7.90 MBytes 8ms
[ 7] 4.00-5.00 sec 2.38 MBytes 19.9 Mbits/sec 1 8.00 MBytes 17ms
[ 7] 5.00-6.00 sec 2.38 MBytes 19.9 Mbits/sec 1 8.00 MBytes 8ms
[ 7] 6.00-7.00 sec 776 KBytes 6.36 Mbits/sec 4 1.16 KBytes 10ms <--network outage start
[ 7] 7.00-8.00 sec 0.00 Byte s 0.00 bits/sec 1 1.16 KBytes 10ms
[ 7] 8.00-9.00 sec 0.00 Byte s 0.00 bits/sec 1 1.39 KBytes 10ms
[ 7] 9.00-10.00 sec 0.00 Byte s 0.00 bits/sec 1 1.39 KBytes 10ms
[ 7] 10.00-11.00 sec 0.00 Byte s 0.00 bits/sec 0 1.39 KBytes 10ms
[ 7] 11.00-12.00 sec 0.00 Byte s 0.00 bits/sec 1 1.39 KBytes 10ms
[ 7] 12.00-13.00 sec 0.00 Byte s 0.00 bits/sec 0 1.39 KBytes 10ms
[ 7] 13.00-14.00 sec 0.00 Byte s 0.00 bits/sec 1 1.39 KBytes 10ms
[ 7] 14.00-15.00 sec 0.00 Byte s 0.00 bits/sec 0 1.39 KBytes 10ms
[ 7] 15.00-16.00 sec 0.00 Byte s 0.00 bits/sec 1 1.39 KBytes 10ms
[ 7] 16.00-17.00 sec 0.00 Byte s 0.00 bits/sec 0 1.39 KBytes 10ms
[ 7] 17.00-18.00 sec 0.00 Byte s 0.00 bits/sec 1 1.39 KBytes 10ms
[ 7] 18.00-19.00 sec 0.00 Byte s 0.00 bits/sec 0 1.39 KBytes 10ms
[ 7] 19.00-20.00 sec 8.37 MBytes 70.2 Mbits/sec 26 1.04 MBytes 20ms <-- network recovered
[ 7] 20.00-21.00 sec 26.7 MBytes 224 Mbits/sec 0 1.08 MBytes 4ms <-- unbounded overshoot
[ 7] 21.00-22.00 sec 2.38 MBytes 19.9 Mbits/sec 0 1.08 MBytes 7ms <-- settle back to average
[ 7] 22.00-23.00 sec 2.38 MBytes 19.9 Mbits/sec 1 1.08 MBytes 8ms
[ 7] 23.00-24.00 sec 2.38 MBytes 19.9 Mbits/sec 3 1.08 MBytes 10ms
[ 7] 24.00-25.00 sec 2.38 MBytes 19.9 Mbits/sec 2 1.09 MBytes 20ms
[ 7] 25.00-26.00 sec 2.38 MBytes 19.9 Mbits/sec 2 1.13 MBytes 9ms
[ 7] 26.00-27.00 sec 2.38 MBytes 19.9 Mbits/sec 3 1.20 MBytes 10ms
[ 7] 27.00-28.00 sec 2.38 MBytes 19.9 Mbits/sec 2 1.28 MBytes 9ms
[ 7] 28.00-29.00 sec 2.38 MBytes 19.9 Mbits/sec 1 1.38 MBytes 7ms
^C[ 7] 29.00-29.28 sec 640 KBytes 18.9 Mbits/sec 0 1.41 MBytes 12ms
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 7] 0.00-29.28 sec 69.8 MBytes 20.0 Mbits/sec 62 sender <-- correct long-term average.
[ 7] 0.00-29.28 sec 0.00 Byte s 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated
```

## Steps to Reproduce

* cause a short network outage between the iperf client and server
* ensure the outage is not directly detectable as link-state drop by the socket on the client or server.
* re-establish the network connectivity
* watch iperf overshoot unbounded at maximum throughput

## Possible Solution

* Documentation - ensure --bitrate option is clear and its not a maximum target, but a long-term average target and the system may burst way over the average target if required, even if burst it no specified.
* Or use a better algorithm that does some form of closed-loop adaptive rate limiting.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.