AdguardTeam / AdguardTeam/AdGuardVPNCLI
VPN_SS_RECOVERING loops indefinitely on half-open TCP connections (SOCKS mode)
- Lingua principale
- Shell
- Stelle
- 209
- Fork
- 15
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
### Please answer the following question for yourself before submitting an issue
- [x] I checked to make sure that this issue has not already been filed
### AdGuard VPN CLI version
1.7.12 (RELEASE channel)
### Environment
* OS: Entware (BusyBox v1.37.0, ash) on a consumer router
* Architecture: MIPS (mipsel), MediaTek MT7621, 256 MB RAM
* Mode: SOCKS5, host 127.0.0.1, no auth
* Protocol: auto (resolves to HTTP/2 in practice — all outbound connections are TCP:443, no UDP observed)
* Subscription: paid
* Setup: three independent CLI instances running in parallel, each with its own HOME
profile directory and SOCKS port (127.0.0.1:1080 / :1081 / :1082)
* Network: router is behind ISP-level CGNAT; the upstream path goes through DPI-based
filtering that can silently blackhole an established flow without sending RST or FIN
### Issue Details
The trigger condition is a blackholed outbound path that does NOT reset the TCP session.
In the wild this happens under DPI throttling; in a lab it is reproducible with a firewall
rule that silently DROPs packets of the established flow (DROP, not REJECT).
Steps:
1. Run the CLI in SOCKS mode and connect to any location.
2. Blackhole the client's outbound path to the entry endpoint without resetting the session
(DPI throttling in the wild / iptables DROP rule in a lab).
3. Observe: the location ping starts failing and the client enters the recovery loop.
4. Wait — the client never leaves the loop on its own (observed for several hours).
5. Run `disconnect`, then `connect` — the tunnel is restored immediately, every time.
Frequency: reproduced on several different days, on different profiles, on different
locations (two European, one North American) and different entry endpoints.
In one incident two of three instances were stuck simultaneously. In another, one instance
was stuck while the other two — connected to the SAME entry IP — kept working normally,
which rules out an outage of that entry node.
### Expected Behavior
After a few consecutive failed pings, the client should tear down the transport
connections and perform a full re-dial (re-selecting the entry endpoint), i.e. do
internally what `disconnect` + `connect` does — and recover without user intervention.
While the tunnel is not actually passing traffic, the SOCKS listener should not silently
accept connections that will hang.
### Actual Behavior
The client enters an infinite VPN_SS_WAITING_RECOVERY -> VPN_SS_RECOVERING loop and never
recovers. `status` keeps reporting "Reconnecting to [LOCATION]", the SOCKS5 listener stays
up and accepts connections that never carry any traffic, so every application behind the
proxy hangs. On a router this stalls the whole LAN until a human intervenes.
Root cause (as far as I can tell from the logs and socket state): the recovery routine
reuses the existing TCP connections to the entry endpoint instead of closing and re-dialing
them. Those sockets are half-open — ESTABLISHED with a stuck non-zero Send-Q — so every
ping attempt is guaranteed to fail.
`vpn_cm_stop` is called on every iteration, but a new `Using endpoint:` line NEVER appears
afterwards: no endpoint is re-selected or re-dialed during recovery. The same sockets persist
in netstat across hundreds of recovery iterations.
### Screenshots
No screenshots — logs and socket state below. (Angle-bracket placeholders in the original post were stripped as HTML; replaced with square brackets.)
**tunnel.log — the loop (repeats every ~5 s, for hours)**
```
WARN [727] VPNCORE pinger_handler: [0] Failed to ping location
INFO [727] VPNCORE raise_state: [0] VPN_SS_WAITING_RECOVERY
INFO [727] VPN_CM vpn_cm_stop: [0] ...
INFO [727] VPN_CM vpn_cm_stop: [0] Done
WARN [727] STANDALONE_CLIENT vpn_handler: Waiting recovery: to next=0ms error=1 Failed to ping location
INFO [727] VPNCORE raise_state: [0] VPN_SS_RECOVERING
INFO [727] VPN_CM vpn_cm_stop: [0] ...
INFO [727] VPN_CM vpn_cm_stop: [0] Done
WARN [727] VPNCORE pinger_handler: [0] Failed to ping location
INFO [727] VPNCORE raise_state: [0] VPN_SS_WAITING_RECOVERY
... (identical cycle, ~5 s apart, indefinitely)
```
For contrast, a healthy connect emits (endpoint masked — it is your infrastructure):
```
INFO [727] VPNCORE pinger_handler: [0] Using endpoint: name=[masked-hostname], address=[entry-ip]:443, relay=none, ping=397ms
```
Line counts matching `error|timeout|fail|reconnect` during one incident: ~1075 in each stuck profile's log, 0 in the healthy one.
**netstat — half-open sockets with stuck Send-Q (key evidence)**
PID 705 = stuck instance; PIDs 737 and 767 = healthy instances connected to the **same** entry IP:
```
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program
tcp 0 295 [wan-ip]:53074 [entry-ip]:443 ESTABLISHED 705/adguardvpn-cli <-- stuck
tcp 0 295 [wan-ip]:53078 [entry-ip]:443 ESTABLISHED 705/adguardvpn-cli <-- stuck
tcp 0 0 [wan-ip]:51350 [entry-ip]:443 ESTABLISHED 737/adguardvpn-cli <-- healthy
tcp 0 0 [wan-ip]:51294 [entry-ip]:443 ESTABLISHED 737/adguardvpn-cli <-- healthy
tcp 0 0 [wan-ip]:51316 [entry-ip]:443 ESTABLISHED 737/adguardvpn-cli <-- healthy
tcp 0 0 [wan-ip]:59184 [entry-ip]:443 ESTABLISHED 767/adguardvpn-cli <-- healthy
```
The stuck instance holds sockets with a non-empty send queue — data was written and never acknowledged. From the kernel's point of view the sockets stay ESTABLISHED, so nothing ever signals the client that the path is gone.
An earlier incident produced the same signature on a different day and a different entry endpoint: two instances stuck with Send-Q 287 and 294, the third healthy.
**Client state during the incident**
```
$ HOME=[profile-A] adguardvpn-cli status
Reconnecting to [LOCATION] in SOCKS mode, listening on 127.0.0.1:1080
$ curl -s --max-time 8 -x socks5h://127.0.0.1:1080 ifconfig.me
(empty — no response, hangs until timeout)
```
Fix that always works:
```
HOME=[profile] adguardvpn-cli disconnect
sleep 5
HOME=[profile] adguardvpn-cli connect
# → Successfully Connected, traffic flows immediately
```
### Additional Information
Suggested fixes, in order of impact:
1. Escalate recovery. After N consecutive failed pings (e.g. 3-5), close all transport
sockets and perform a full re-dial, including re-selecting/re-resolving the entry
endpoint. Ideally with backoff, and preferably picking a different entry endpoint after
repeated failures.
2. Detect dead sockets. Enable TCP keepalive with an aggressive timeout, and/or set
SO_SNDTIMEO / TCP_USER_TIMEOUT on transport sockets, so a blackholed path surfaces as an
error instead of an indefinitely ESTABLISHED socket. A persistently non-zero Send-Q is a
reliable local indicator of this state.
3. Do not accept SOCKS connections while the tunnel is not connected (or make it
configurable). Today the listener accepts clients during RECOVERING, so applications hang
instead of failing fast and falling back.
4. Surface the state in the exit code. A non-zero exit code from `status` when the tunnel is
not actually passing traffic would let scripts detect this without an external probe.
Current workaround — an external watchdog that probes end-to-end traffic through each SOCKS
port and forces a reconnect on failure:
#!/bin/sh
export SSL_CERT_FILE=/opt/etc/ssl/certs/ca-certificates.crt
for prof in profileA:1080 profileB:1081 profileC:1082; do
h=/opt/home/${prof%%:*}
p=${prof##*:}
curl -s --max-time 15 -o /dev/null -x socks5h://127.0.0.1:$p https://example.com && continue
HOME=$h /opt/adguardvpn_cli/adguardvpn-cli disconnect >/dev/null 2>&1
sleep 3
HOME=$h /opt/adguardvpn_cli/adguardvpn-cli connect
done
Note that a port-liveness check (netstat/ss/nc) is useless here: the SOCKS port stays open
the whole time. Only an end-to-end probe detects the condition.
Notes:
* Debug logging was OFF during these incidents; the excerpts above come from the standard
tunnel.log. I can reproduce with `set-debug-logging on` and provide full logs, including
the exact entry endpoints and timestamps, privately — I masked them here since they are
part of your infrastructure. Just tell me where to send them.
* Possibly worth a separate issue: on a filtered network, direct calls to the AdGuard backend
regularly time out (`Failed to get auth link` on login, `rc=13 Unable to fetch locations` on
connect, `WARN BackendConfigFetcher get_config: Backend config was not fetched, continue
with cached or default`), while the tunnel protocol itself connects fine. Routing the
backend domains through an already-established tunnel works around it. Built-in retries and
a fallback path for backend requests at startup would help a lot — right now a failed
backend call at boot leaves the client refusing to connect at all, even though the tunnel
transport would work.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.