AdguardTeam / AdguardTeam/AdguardForWindows

Regression in 8.0 RC 3 (5542) SockFilter: 10GbE receive throughput capped at ~3.7 Gbit/s with heavy TCP retransmissions

Ouverte
#6,127 9 commentaires 2 réactions 1 personne assignée Réclamée par @AlexandrPkhm Voir sur GitHub
Bug Priority: P4
Langage dominant
Aucune donnée de langage
Étoiles
887
Forks
84
Merge moyen
27 min
PR mergées (30 j)
2

Description

### Please answer the following questions for yourself before submitting an issue

- [x] Filters were updated before reproducing an issue
- [x] I checked the [knowledge base](https://adguard.com/kb/) and found no answer
- [x] I checked to make sure that this issue has not already been filed

### AdGuard version

8.0.0 RC 3 (5542)

### Browser version

Not applicable

### OS version

Windows 11 IoT Enterprise LTSC 2024, Version 2009 (OS Build 26100)

### Traffic filtering

- [x] yes, I do

### Ad Blocking

_No response_

### Privacy

_No response_

### Social

_No response_

### Annoyances

_No response_

### Security

_No response_

### Other

_No response_

### Language-specific

_No response_

### Which DNS server do you use?

AdGuard DNS

### DNS protocol

None

### Custom DNS

_No response_

### What Stealth Mode options do you have enabled?

_No response_

### Support ticket ID

_No response_

### Issue Details

This is the second of two defects found while investigating #6120. It is filed separately because it is independent of the RSC and WFP callout behaviour tracked there, it affects a different kernel driver, and it is a regression in an unreleased RC rather than long-standing behaviour.

Summary: with AdGuard 8.0.0 RC 3 (5542) in SockFilter mode, 10GbE receive throughput on Windows is capped at roughly 3.7 Gbit/s regardless of how many parallel TCP streams are used, accompanied by roughly 8,500 to 8,700 TCP retransmissions per 15 second test. AdGuard 7.22.9 in SockFilter mode on the same machine reaches 9.31 Gbit/s with zero retransmissions.

## Environment

Windows host (the receiver in the failing direction):

- Dell OptiPlex 7040, Intel Core i7-6700 @ 3.40 GHz, 4 physical cores, 8 threads, 64 GB RAM
- Windows 11 IoT Enterprise LTSC 2024, build 26100
- Mellanox ConnectX-3 Ethernet Adapter, mlx4eth63.sys 5.50.14695.0
- 10 Gbps full duplex over DAC, MTU 1500, RSS enabled with 8 receive queues
- AdGuard kernel driver build 8.0.87.0 in both filtering modes

Peer: Linux host with Mellanox ConnectX-3, mlx4_en, firmware 2.36.5150. Same switch, same cable, unchanged throughout.

Measurement: iperf3, 15 second runs, port 5201. Without -R the Linux host sends and Windows receives.

## Steps to reproduce

1. Install AdGuard for Windows 8.0.0 RC 3 (5542) on a machine with a 10GbE adapter and set the filtering driver to SockFilter.
2. Confirm `adgnetworksockdrv` is RUNNING and `adgnetworkwfpdrv` is not installed.
3. Confirm RSC is operational: `Get-NetAdapterRsc` reports IPv4Enabled True, IPv4OperationalState True, IPv4FailureReason NoFailure.
4. From a 10GbE peer, run `iperf3 -c -P 1 -t 15` and then `iperf3 -c -P 8 -t 15`.
5. Repeat with the filtering driver set to WFP for comparison.

## Measurements
| Config | RSC operational | P1 (Gbit/s) | P8 (Gbit/s) | P8 retrans | P8/P1 |
|---|---|---|---|---|---|
| no AdGuard driver loaded | True | 5.95 | 9.25 | 0 | 1.55x |
| 7.22.9 SockFilter | True | 5.03 | 9.31 | 0 | 1.85x |
| 8.0 RC3 WFP | False (WFPCompatibility) | 3.0 - 3.3 | 8.5 - 9.1 | 0 - 1 | 2.8x |
| 8.0 RC3 SockFilter | True | 3.2 - 3.6 | 3.6 - 3.9 | 8541 - 8698 | 1.1x |

Reverse direction (Windows sending) reaches approximately 9.2 Gbit/s in every configuration, so the problem is confined to the Windows receive path.

The RC3 SockFilter result was reproduced three times, twice without Debug logging and once with it, giving 3.67, 3.90 and 3.63 Gbit/s at P8. Debug logging is not a factor.

## Analysis

Sorting the results by scaling behaviour rather than by RSC state is what makes the pattern visible. Every healthy configuration gains substantially from additional streams and loses no packets. RC3 SockFilter gains almost nothing: 3.4 Gbit/s with one stream, 3.7 Gbit/s with eight.

That combination is characteristic of a single global serialisation point in the receive path rather than a per-flow or per-core limit:

- At P1 the offered load is close to the ceiling, so almost nothing is dropped (0 to 309 retransmissions).
- At P8 the offered load is well above the ceiling, throughput does not increase, and the excess is discarded (8541 to 8698 retransmissions).
- Retransmissions are spread evenly across the full 15 second run at roughly 580 per second and across all eight streams (879 to 1230 each). This is not a connection setup problem and not one pathological flow.

Neither CPU capacity, RSS nor the NIC can be the limiting factor, because the same machine with the same NIC and the same RSS configuration reaches 9.13 Gbit/s at P8 in WFP mode. Note also that RC3 WFP achieves this with RSC not operational, so it is doing strictly more per-packet work in the receive path than SockFilter, which has RSC available.

Single-stream performance also regressed, from 5.03 Gbit/s under 7.22 SockFilter to about 3.4 Gbit/s under RC3 SockFilter, with RSC operational in both cases. Interestingly that is close to the RC3 WFP single-stream figure of 3.0 to 3.3 Gbit/s, which is measured with RSC not operational. This raises the question of whether RSC is still effectively coalescing under RC3 SockFilter, since `IPv4OperationalState: True` is a state flag and not a measurement of actual coalescing.

## What the debug logs show

Debug logs for both driver modes were captured and sent to qa@adguard.com referencing #6120.

TCP port 5201 does not appear anywhere in the agent or the service log, in either driver mode. The service log line rate stays constant at 22 to 66 lines per second across the entire test window, with no change when iperf3 starts or stops, and no related warnings or errors.

The user-mode service (AGNFHandler, AGProxySession, AGHttpProcessingUnit) therefore does not appear to be involved. The behavioural difference is in the kernel drivers.

## Control

A diff of the two configuration captures taken during the SockFilter and WFP reproductions shows exactly one functional difference: `Driver Type: Sock` versus `Driver Type: Wfp`. HTTPS filtering enabled, `UseRedirectDriver: false`, `FilterLocalhost: true`, AntiDPI disabled and all Stealth Mode settings are identical. Same NICs, cable, switch, MTU and peer host throughout.

## Possible relationship to 7.22 WFP

AdGuard 7.22.9 in WFP mode shows the same signature in a more severe form: 1.62 Gbit/s at P1 and 1.44 Gbit/s at P8, so it was actually slower with eight streams than with one, with 2354 retransmissions. Negative scaling is a lock convoy signature rather than CPU saturation.

The 8.0 WFP path appears to have resolved this, while the 8.0 SockFilter path appears to have introduced or reintroduced it. It may be worth checking whether both paths share the same receive pipeline code.

## What is not yet established

- Whether the retransmissions represent real packet loss or spurious RTOs caused by long stalls. The iperf3 Retr value is a sender-side counter. Linux-side DSACK counters would settle this.
- Whether RSC is actually coalescing under RC3 SockFilter. `Get-NetAdapterRscStatistics` was not captured.
- The exact internal location of the drop or stall. No per-core CPU, DPC or NIC discard counters were collected.

## Suggested first checks

In the `adgnetworksockdrv` receive path, look for a single global serialisation point introduced since 7.22: one worker thread queue, one lock around packet processing, a shared connection table with an exclusive lock, or a fixed-depth queue with a silent tail drop. If there is no counter for discarded packets in that path, adding one would answer this question immediately.

Cross-reference: #6120 tracks the separate WFP and RSC callout flag behaviour. The two are independent, shown by a clean 2x2: RC3 WFP exhibits the RSC issue without this regression, and RC3 SockFilter exhibits this regression without the RSC issue.

### Expected Behavior

AdGuard 8.0 RC 3 in SockFilter mode should sustain approximately the same 10GbE receive throughput as AdGuard 7.22.9 in SockFilter mode on the same hardware, which was 9.31 Gbit/s with 8 parallel TCP streams and zero retransmissions, and it should scale with additional parallel streams.

### Actual Behavior

Aggregate receive throughput is capped at approximately 3.7 Gbit/s and does not increase with additional parallel streams (3.4 Gbit/s with one stream, 3.7 Gbit/s with eight). Load offered above that ceiling is discarded, producing 8541 to 8698 TCP retransmissions per 15 second test, spread evenly across the full run and across all eight streams. Single-stream throughput also dropped from 5.03 Gbit/s under 7.22.9 to approximately 3.4 Gbit/s, with RSC operational in both cases.

### Screenshots

Screenshot 1

### Additional Information

Sanitized Debug logs for both driver modes were emailed to qa@adguard.com with [windows] and 6120 in the subject. Each archive covers the exact reproduction window and includes the driver installation and startup lines, a configuration comparison of the two runs, and a hardware and OS summary. Happy to run additional tests, including pktmon drop attribution with --drop-only --comp all, per-core DPC and Per Processor Network Interface Card Activity measurements, a P1/P2/P4/P8 scaling curve, Get-NetAdapterRscStatistics deltas, and Linux-side DSACK counters.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.