cloudflare / cloudflare/boringtun
Intermittent connection loss with HANDSHAKE(REKEY_TIMEOUT) errors
- Dominant language
- Rust
- Stars
- 7.2k
- Forks
- 532
- PR merge metrics
- No merged PRs in 30d
Description
Preface:
- running warp client but i think this is an issue with boringtun under the hood
Setup:
- boringtun as part of the warp client on Rocky 9.2\
- cloudflare zero-trust with local fallback domain and single tunnel
- split tunnel in exclude mode with nothing excluded (everything routes through the single tunnel)
- cloudflare authentication using SSO (via google workspace SAML)
- warp-cli version 2023.7.40
Issue:
I connect the warp client successfully and everything works (routes to my private network as well as internet routes). Eventually, sometimes minutes sometimes hours, all network connectivity dies. In this state I cannot access anything, including DNS and internet. In the below snippet of boringtun.log, the connectivity dies around 14:05:32 (timestamp collected by pinging google.com every second until the issue is observed).
```
2023-09-09T13:59:37.890Z DEBUG boringtun::noise: Sending keepalive
2023-09-09T13:59:39.377Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:00:03.377Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:00:27.676Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:00:37.877Z DEBUG boringtun::noise::timers: SESSION_EXPIRED(REJECT_AFTER_TIME) session=6252035 index=24422
2023-09-09T14:00:51.976Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:01:15.977Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:01:38.176Z DEBUG boringtun::noise::timers: HANDSHAKE(REKEY_AFTER_TIME (on send))
2023-09-09T14:01:38.176Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:01:38.188Z DEBUG boringtun::noise: Received handshake_response local_idx=6252037 remote_idx=6836995
2023-09-09T14:01:38.189Z DEBUG boringtun::noise: New session session=6252037 index=24422
2023-09-09T14:01:38.189Z DEBUG boringtun::noise: Sending keepalive
2023-09-09T14:01:40.276Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:02:04.576Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:02:28.576Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:02:38.177Z DEBUG boringtun::noise::timers: SESSION_EXPIRED(REJECT_AFTER_TIME) session=6252036 index=24422
2023-09-09T14:02:52.876Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:03:17.176Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:03:38.177Z DEBUG boringtun::noise::timers: HANDSHAKE(REKEY_AFTER_TIME (on send))
2023-09-09T14:03:38.178Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:03:38.191Z DEBUG boringtun::noise: Received handshake_response local_idx=6252038 remote_idx=6836996
2023-09-09T14:03:38.192Z DEBUG boringtun::noise: New session session=6252038 index=24422
2023-09-09T14:03:38.192Z DEBUG boringtun::noise: Sending keepalive
2023-09-09T14:03:41.177Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:04:05.476Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:04:29.476Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:04:38.176Z DEBUG boringtun::noise::timers: SESSION_EXPIRED(REJECT_AFTER_TIME) session=6252037 index=24422
2023-09-09T14:04:53.477Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:05:14.176Z WARN boringtun::noise::timers: HANDSHAKE(KEEPALIVE + REKEY_TIMEOUT)
2023-09-09T14:05:14.177Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:05:19.276Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:05:19.277Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:05:24.377Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:05:24.377Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:05:29.477Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:05:29.478Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:05:34.577Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:05:34.577Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:05:39.677Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:05:39.678Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:05:44.777Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:05:44.777Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:05:49.877Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:05:49.877Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:05:54.976Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:05:54.977Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:00.076Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:06:00.076Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:05.177Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:06:05.177Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:10.276Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:06:10.277Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:15.376Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:06:15.377Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:20.476Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:06:20.477Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:25.577Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:06:25.577Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:30.677Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:06:30.677Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:35.776Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:06:35.777Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:38.477Z DEBUG boringtun::noise::timers: SESSION_EXPIRED(REJECT_AFTER_TIME) session=6252038 index=24422
2023-09-09T14:06:40.877Z WARN boringtun::noise::timers: HANDSHAKE(REKEY_TIMEOUT)
2023-09-09T14:06:40.877Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:44.177Z ERROR boringtun::noise::timers: CONNECTION_EXPIRED(REKEY_ATTEMPT_TIME)
2023-09-09T14:06:44.237Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:44.238Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:06:44.972Z DEBUG boringtun::noise: Received handshake_response local_idx=13336833 remote_idx=12028672
2023-09-09T14:06:44.973Z DEBUG boringtun::noise: New session session=13336833 index=52097
2023-09-09T14:06:44.973Z DEBUG boringtun::noise: Sending keepalive
2023-09-09T14:07:08.422Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:07:32.722Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:07:56.723Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:08:21.022Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:08:44.423Z DEBUG boringtun::noise::timers: HANDSHAKE(REKEY_AFTER_TIME (on send))
2023-09-09T14:08:44.424Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:08:44.436Z DEBUG boringtun::noise: Received handshake_response local_idx=13336834 remote_idx=12028673
2023-09-09T14:08:44.437Z DEBUG boringtun::noise: New session session=13336834 index=52097
2023-09-09T14:08:44.437Z DEBUG boringtun::noise: Sending keepalive
2023-09-09T14:08:45.023Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:09:09.023Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:09:33.323Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:09:44.422Z DEBUG boringtun::noise::timers: SESSION_EXPIRED(REJECT_AFTER_TIME) session=13336833 index=52097
2023-09-09T14:09:57.622Z DEBUG boringtun::noise::timers: KEEPALIVE(PERSISTENT_KEEPALIVE)
2023-09-09T14:10:09.022Z WARN boringtun::noise::timers: HANDSHAKE(KEEPALIVE + REKEY_TIMEOUT)
2023-09-09T14:10:09.023Z DEBUG boringtun::noise: Sending handshake_initiation
2023-09-09T14:10:09.140Z DEBUG boringtun::noise: Received handshake_response local_idx=13336835 remote_idx=15038208
```
Contributor guide
Assessment
This issue has not been assessed yet.