SagerNet / SagerNet/sing-box

[Bug] iOS NetworkExtension critical memory pressure after short-flow churn in system stack

Open
#4,469 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
38.1k
Forks
4.6k
Avg merge
19d 15h
Merged PRs (30d)
1

Description

Operating system

iOS

System version

iOS 27.0 (24A5424a), iPhone 16 (iPhone17,3)

Installation type

sing-box for iOS Graphical Client

If you are using a graphical client, please provide the version of the client.

SagerNet/sing-box-for-apple@ebc76e9eaa347a2dd6222583b72f4e8ffa9741cf, Release build

Version
sing-box 1.14.0-rc.1 (8dd67a1e49711ce8a9a884bef60a2139ef36446f)
Go go1.26.6 ios/arm64
sing-tun v0.9.0-beta.3
Description

An iOS Packet Tunnel using the system stack received critical memory pressure after approximately 91,493 seconds (25 h 25 min) of uptime.

The OOM snapshot was generated by the NetworkExtension memory-pressure callback. There was no new Go panic, native crash report, or As4 called on IPv6 address failure. A corresponding Jetsam snapshot identified the installed Packet Tunnel binary by UUID and showed it active at about 42.5 MiB with a lifetime maximum of about 49.8 MiB; it was not the victim of that Jetsam event. Thus this report is about persistent memory pressure near the iOS NetworkExtension limit, not a confirmed process kill.

Snapshot metadata:

memory usage:     40 MB
available memory: 8.1 MB
HeapAlloc:        17 MB
HeapInuse:        22 MB
HeapSys:          36 MB
StackInuse:       3.8 MB
Sys:              48 MB
goroutines:       103
active connections reported by sing-box: 9 (6 TCP, 3 UDP)

The retained heap profile is dominated by the system-stack NAT/flow data structures:

Type: inuse_space
Total sampled heap: 14.18 MiB

4.32 MiB  github.com/sagernet/sing-tun.(*TCPNat).Lookup
1.63 MiB  github.com/sagernet/sing-tun.(*ForwardDispatcher).insertEntry
0.50 MiB  github.com/sagernet/sing-tun.(*ForwardDispatcher).installSimple
          (2.13 MiB cumulative through installSimple)

The inuse_objects profile also attributes a sampling-weight estimate of 16,384 objects to ForwardDispatcher.installSimple. This is not treated as an exact live-entry count, but it coincides with flowTableCapacity = 16384 and is consistent with a high-water allocation in the flow table.

This does not match #4312: that report had 936 goroutines retained by DNS packet connections. This snapshot has 103 goroutines and no large repeated DNS goroutine stacks. A burst of app traffic preceded the pressure callback, but the retained profile points to the NAT and flow maps rather than a goroutine leak.

The same TCP NAT map implementation is still present in sing-tun v0.9.0-beta.4 used by sing-box 1.14.0-rc.5.

Expected behavior: after a short-flow burst subsides and sessions expire or the NAT is purged, the system stack should be able to return close to its pre-burst memory footprint, particularly under the approximately 50 MiB iOS NetworkExtension limit.

Reproduction

This source-level reproduction is local and does not require a remote server, graphical client, TUN interface, or privileged networking.

git clone https://github.com/SagerNet/sing-tun.git
cd sing-tun
git checkout v0.9.0-beta.4
# Save the test below as tcp_nat_memory_repro_test.go
go test . -run '^TestTCPNatRetainedMemory$' -count=1 -v
package tun

import (
    "context"
    "net/netip"
    "runtime"
    "testing"
    "time"
)

func TestTCPNatRetainedMemory(t *testing.T) {
    const flowCount = 20_000

    runtime.GC()
    var before runtime.MemStats
    runtime.ReadMemStats(&before)

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    nat := NewNat(ctx, 5*time.Minute)
    sourceAddress := netip.MustParseAddr("172.19.0.2")
    for index := range flowCount {
        source := netip.AddrPortFrom(sourceAddress, uint16(10_000+index))
        destinationAddress := netip.AddrFrom4([4]byte{
            198, 18, byte(index >> 8), byte(index),
        })
        destination := netip.AddrPortFrom(destinationAddress, 443)
        if port := nat.Lookup(source, destination); port == 0 {
            t.Fatalf("TCP NAT port allocation failed at flow %d", index)
        }
    }

    runtime.GC()
    var afterFill runtime.MemStats
    runtime.ReadMemStats(&afterFill)
    fillDelta := int64(afterFill.HeapAlloc) - int64(before.HeapAlloc)

    nat.Purge()
    runtime.GC()
    var afterPurge runtime.MemStats
    runtime.ReadMemStats(&afterPurge)
    purgeDelta := int64(afterPurge.HeapAlloc) - int64(before.HeapAlloc)
    t.Logf("%d distinct TCP flows: %.2f MiB retained before timeout; %.2f MiB still retained after Purge", flowCount, float64(fillDelta)/(1024*1024), float64(purgeDelta)/(1024*1024))
    runtime.KeepAlive(nat)
}

Output on v0.9.0-beta.4:

=== RUN   TestTCPNatRetainedMemory
    tcp_nat_memory_repro_test.go:46: 20000 distinct TCP flows: 4.90 MiB retained before timeout; 3.07 MiB still retained after Purge
--- PASS: TestTCPNatRetainedMemory (0.01s)

TCPNat.Purge currently calls clear on addrMap and portMap. This removes the entries but retains the Go map bucket high-water capacity. As a validation experiment only, replacing those two clear calls with new empty maps reduced post-purge retained heap from 3.07 MiB to 0.01 MiB in the same test:

n.addrMap = make(map[tcpNatKey]uint16)
n.portMap = make(map[uint16]*TCPSession)

Purge alone may not be sufficient for a long-running NetworkExtension because ordinary timeout cleanup also deletes entries without shrinking the maps. A periodic rebuild after a large drop in live entries, or another bounded/shrinkable table strategy, may be needed. ForwardDispatcher.table appears to have a similar high-water characteristic.

Logs
WARN[91493] service/oom-killer[0]: memory pressure: critical, usage: 40 MB

File: Extension
Type: inuse_space
Time: 2026-08-31 09:03:02 CST
Showing nodes accounting for 14.18 MiB total sampled heap
4.32 MiB  github.com/sagernet/sing-tun.(*TCPNat).Lookup
1.63 MiB  github.com/sagernet/sing-tun.(*ForwardDispatcher).insertEntry
0.50 MiB  github.com/sagernet/sing-tun.(*ForwardDispatcher).installSimple


Full heap and goroutine profiles can be provided if requested. The full client configuration is not included because it contains proxy credentials and is not required by the source-level reproduction above.
Supporter
Integrity requirements
  • I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
  • I confirm that I have provided the server and client configuration files and process that can be reproduced locally, instead of a complicated client configuration file that has been stripped of sensitive data.
  • I confirm that I have provided the simplest configuration that can be used to reproduce the error I reported, instead of depending on remote servers, TUN, graphical interface clients, or other closed-source software.
  • I confirm that I have provided the complete configuration files and logs, rather than just providing parts I think are useful out of confidence in my own intelligence.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in sing-tun's TCPNat.Lookup and Purge implementations, then inspect ForwardDispatcher.table for the related high-water behavior. Run the provided TestTCPNatRetainedMemory reproduction on v0.9.0-beta.4 and compare retained heap before and after purge. Done means short-flow churn and cleanup no longer leave memory near the iOS NetworkExtension limit, with regression coverage for the observed retention.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.