SagerNet / SagerNet/sing-box

fatal error: thread exhaustion after 30-60min uptime (goroutine leak in udpnat2 NewPacket), 1.13.19 Windows

Open
#4,446 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
38.1k
Forks
4.6k
Avg merge
19d 15h
Merged PRs (30d)
1

Description

Operating system

Windows

System version

Windows 11 Pro 26200.9168

Installation type

Third-party graphical clients that advertise themselves as using sing-box (Windows)

If you are using a graphical client, please provide the version of the client.

GUI.for.SingBox (latest, version tag not exposed in-app), https://github.com/GUI-for-Cores/GUI.for.SingBox

Version
sing-box version 1.13.19
Environment: go1.26.5 windows/amd64
Tags: with_gvisor,with_quic,with_dhcp,with_wireguard,with_utls,with_acme,with_clash_api,with_tailscale,with_ccm,with_ocm,with_naive_outbound,with_purego,badlinkname,tfogo_checklinkname0
Revision: b5ebaa1fc0f2b94256180b95468e73ef53caa27d
CGO: disabled
Description

Environment

  • sing-box 1.13.19 stable, Windows 11 Pro (build 26200.9168), managed by third-party GUI GUI.for.SingBox (GUI only starts/stops the core and generates the config; crashes reproduced across GUI restarts)
  • TUN inbound: stack: mixed, auto_route: true, strict_route: true, endpoint_independent_nat: false
  • DNS: fakeip (198.18.0.0/15) for A/AAAA + DoH upstreams (remote DoH via proxy detour, local DoH with UDP bootstrap), hijack-dns action
  • ~153 outbounds (88 trojan, 33 vless, urltest groups...), 27 route rules on rule_sets

Symptom

The core dies every 30-90 minutes under normal desktop traffic with fatal error: thread exhaustion.

Telemetry (process thread count sampled every 10s): starts at ~40 threads, climbs monotonically to 120+ within ~30 min and never drops, until the process aborts. Crash intervals observed: 64, 36, 21 minutes.

At the last crash the log file had grown to 851,886 lines: the tail is a full Go runtime goroutine dump, and the overwhelming majority of goroutines end with:

created by github.com/sagernet/sing/common/udpnat2.(*Service).NewPacket.func1 in goroutine NNN
    github.com/sagernet/sing@v0.8.13/common/udpnat2/service.go:67 +0x435

So each UDP NAT session spawned from udpnat2.NewPacket appears to leak a goroutine (and eventually an OS thread), never released.

Possibly related: #4312 (closed, same fatal error, iOS alpha) and #4428 (open). My case is the Windows stable 1.13.19 build.

Reproduction
  1. Run sing-box 1.13.19 on Windows with TUN (stack: mixed, strict_route: true) + fakeip DNS + hijack-dns, behind a desktop workload producing many short-lived UDP flows (QUIC/WebRTC/games; in my case Windows Delivery Optimization flooding UDP port 7680 was one major source, but the leak/crash still reproduces after disabling Delivery Optimization).
  2. Sample the process: Get-Process sing-box | Select-Object @{n='threads';e={$_.Threads.Count}} every 10s - thread count grows monotonically from ~40 and never decreases, even when UDP traffic stops.
  3. After ~30-60 minutes the process aborts with fatal error: thread exhaustion and writes a full runtime goroutine dump (hundreds of thousands of lines) to the log before dying.

The crash is load-dependent: heavier UDP churn => faster exhaustion (21-64 min observed).

Logs
Last lines before death (recovered excerpt; the full log was 851,886 lines of Go runtime dump, truncated on rotation - I can provide the raw dump next time it happens if needed):


FATAL> fatal error: thread exhaustion


Recurring goroutine creator from the runtime dump tail (repeated for the vast majority of goroutines):


runtime.goexit({})
    runtime/asm_amd64.s:1771 +0x1 fp=0x... sp=0x... pc=0x...
created by github.com/sagernet/sing/common/udpnat2.(*Service).NewPacket.func1 in goroutine 348
    github.com/sagernet/sing@v0.8.13/common/udpnat2/service.go:67 +0x435


Thread-count telemetry around a typical run (ts, pid, threads, handles, cpu%):


2026-08-25 15:20:50  pid 89976 started with  ~40 threads
2026-08-25 15:48     121 threads (climbing, no release)
2026-08-25 16:01:56,89976,121,1321,53   <- still 121, crash expected before long


<details><summary>Sanitized config (DNS/TUN section, structure identical to the one in use)</summary>


{
 "log": {
  "level": "warn",
  "timestamp": true
 },
 "dns": {
  "servers": {
   "Remote-DNS": {
    "type": "https",
    "server": "***.example.com",
    "domain_resolver": "Local-DNS"
   },
   "Local-DNS": {
    "type": "udp",
    "server": "***"
   },
   "dnsfallback": {
    "type": "https",
    "server": "***.example.com",
    "domain_resolver": "Local-DNS"
   }
  },
  "rules": [
   {
    "query_type": [
     "A",
     "AAAA"
    ],
    "rewrite_dns": true
   },
   {
    "clash_mode": "Global",
    "server": "Remote-DNS"
   },
   {
    "clash_mode": "Direct",
    "server": "Local-DNS"
   },
   {
    "rule_set": "GeoSite-CN",
    "server": "Local-DNS"
   },
   {
    "rule_set": "GeoLocation-!CN",
    "server": "Remote-DNS"
   },
   {
    "outbound": "any",
    "server": "dnsfallback"
   }
  ],
  "final": "Remote-DNS",
  "strategy": "ipv4_only"
 },
 "inbounds": [
  {
   "type": "tun",
   "tag": "tun-in",
   "address": [
    "172.18.0.1/30"
   ],
   "mtu": 0,
   "auto_route": true,
   "strict_route": true,
   "stack": "mixed"
  },
  {
   "type": "mixed",
   "tag": "mixed-in",
   "listen": "127.0.0.1",
   "listen_port": 20122
  }
 ],
 "outbounds": [
  {
   "type": "selector",
   "tag": "proxy",
   "outbounds": [
    "trojan-1",
    "trojan-2",
    "direct"
   ],
   "interrupt_exist_connections": false
  },
  {
   "type": "trojan",
   "tag": "trojan-1",
   "server": "***.example.com",
   "server_port": 443,
   "password": "***"
  },
  {
   "type": "trojan",
   "tag": "trojan-2",
   "server": "***.example.com",
   "server_port": 443,
   "type_new": true,
   "password": "***"
  },
  {
   "type": "direct",
   "tag": "direct"
  }
 ],
 "route": {
  "rules": [
   {
    "action": "sniff"
   },
   {
    "protocol": "dns",
    "action": "hijack-dns"
   },
   {
    "ip_is_private": true,
    "outbound": "direct"
   },
   {
    "rule_set": [
     "GeoSite-CN",
     "GeoLocation-!CN"
    ],
    "action": "resolve"
   },
   {
    "rule_set": "GeoSite-CN",
    "outbound": "direct"
   },
   {
    "rule_set": "GeoLocation-!CN",
    "outbound": "proxy"
   }
  ],
  "auto_detect_interface": true,
  "final": "proxy"
 }
}

</details>
Supporter
Integrity requirements
  • I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
  • I confirm that I have provided the server and client configuration files and process that can be reproduced locally, instead of a complicated client configuration file that has been stripped of sensitive data.
  • I confirm that I have provided the simplest configuration that can be used to reproduce the error I reported, instead of depending on remote servers, TUN, graphical interface clients, or other closed-source software.
  • I confirm that I have provided the complete configuration files and logs, rather than just providing parts I think are useful out of confidence in my own intelligence.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at github.com/sagernet/sing/common/udpnat2/service.go, especially (*Service).NewPacket and the goroutine created at line 67. Reproduce with the Windows TUN, mixed stack and short-lived UDP workload described in the issue, then trace how those goroutines are released when UDP sessions end. Done means the thread count no longer grows under sustained UDP churn and the reported thread-exhaustion failure is no longer reproduced.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.