OpenVPN / OpenVPN/openvpn

Management interface: epoll busy loop when remote client dies without closing connection

Open
#990 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
14.6k
Forks
3.4k
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

When a remote management interface client dies (e.g. crash, power loss) without cleanly closing its TCP
connection, the OpenVPN server process enters a busy loop consuming 100% of a CPU core. The process spins
indefinitely in an epoll_wait / recvfrom cycle on the dead management connection's file descriptor.

The root cause is that the management connection socket enters an inconsistent state where epoll_wait
(level-triggered EPOLLIN) reports readiness, but recvfrom returns EAGAIN. Since OpenVPN uses
level-triggered epoll, this creates an infinite loop:

epoll_wait(9, [{events=EPOLLIN, data={u32=4, u64=4}}], 205, 10000) = 1
recvfrom(10, 0x7ffe05885ec0, 256, MSG_NOSIGNAL, NULL, NULL) = -1 EAGAIN
epoll_wait(9, [{events=EPOLLIN, data={u32=4, u64=4}}], 205, 10000) = 1
recvfrom(10, 0x7ffe05885ec0, 256, MSG_NOSIGNAL, NULL, NULL) = -1 EAGAIN
... (~18,800 iterations/sec)

This continues until the service is manually restarted.

To Reproduce

  1. Configure OpenVPN server with management 0.0.0.0 <port> (externally accessible)
  2. Connect a management client from a remote host
  3. Poweroff the client(not use command, click button) or unplug the network cable
  4. Observe the OpenVPN server process consuming 100% CPU on one core

The issue persists until the service is restarted. System TCP keepalive (tcp_keepalive_time defaults to 7200s)
does not resolve the socket state quickly enough, and even after keepalive would have fired, the socket may
remain in the broken state.

Expected behavior

OpenVPN should handle EAGAIN from recvfrom on the management socket gracefully — either by:

  • Removing the fd from the epoll interest set after repeated EAGAIN and closing the stale connection, or
  • Using SO_KEEPALIVE with reasonable timeouts on management connections to detect dead peers, or
  • Providing a management-timeout configuration option to allow administrators to set an idle timeout for
    management connections

Version information

  • OS: Ubuntu 24.04.1 LTS (Noble Numbat), kernel 6.8.0-51-generic
  • OpenVPN version: 2.6.12 (openvpn 2.6.12-0ubuntu0.24.04.3)
  • OpenVPN build: x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
  • OpenSSL: 3.0.13

Additional context

strace -c output over a 3-second sample during the busy loop:

  % time     seconds  usecs/call     calls    errors syscall
  --- 51.05    0.266354           4     56291           epoll_wait
   48.95    0.255355           4     56291     56291 recvfrom
  ---100.00    0.521709           4    112582     56291 total

ss output showing the problematic socket (fd 10) plus two additional stale CLOSE-WAIT connections on the
management port:

  State      Recv-Q Send-Q   Local Address:Port    Peer Address:Port Process
  LISTEN     2      1              0.0.0.0:7507         0.0.0.0:*     users:(("openvpn",pid=1783203,fd=3))
  CLOSE-WAIT 8      0      ********:7507   ********:51738
  CLOSE-WAIT 8      0      ********:7507   ********:57004
  ESTAB      0      0       ********:7507   ********:48060 users:(("openvpn",pid=1783203,fd=10))

The server had been running for 407 days and is a single-vCPU VM, so the busy loop effectively made the entire
server unresponsive. Restarting openvpn@server-tcp immediately resolved the issue (CPU idle went from 0% to
91%).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the management-interface socket handling around epoll_wait and recvfrom, using the reproduction steps and strace output in this issue. Reproduce the dead-client case and verify that the server no longer spins at 100% CPU and handles the stale connection without requiring a restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.