daeuniverse / daeuniverse/dae

[Bug Report] Improve AnyTLS session lifecycle, observability, and stream correctness in daed

Open
#984 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.2k
Forks
404
Avg merge
57m
Merged PRs (30d)
2

Description

### Checks

- [x] I have searched the existing issues
- [x] I have read the documentation
- [ ] Is it your first time sumbitting an issue

### Current Behavior

Summary

The current daed AnyTLS path is built on top of github.com/daeuniverse/outbound/protocol/anytls rather than sing-anyt
ls. In long-running router workloads, this implementation shows several lifecycle and correctness issues around sessi
on reuse, stream teardown, runtime observability, and multiplexed stream behavior.

Main issues

1. Session lifecycle problems
- Session close/teardown was not robust enough and could leave underlying resources alive longer than expected.
- The outer dialer shutdown path did not fully propagate closure to underlying AnyTLS sessions.
- Idle sessions did not have a complete lifecycle model, which made long-running behavior harder to control.

2. Idle session reuse without health validation
- Idle sessions could be reused without checking whether the underlying tunnel was still healthy.
- This allows half-dead sessions to survive in the idle pool and fail only after being reused by real traffic.

3. Missing runtime diagnostics
- There was no direct runtime visibility into:
- number of AnyTLS dialers
- number of sessions
- number of idle sessions
- number of active streams
- This makes leak diagnosis depend on indirect OS-level observations such as RSS, FD count, and socket state.

4. Stream-layer correctness issues
- The AnyTLS frame header uses a uint16 payload length, but large PSH payloads were not properly bounded or split.
- packetStream.ReadFrom() could leave unread payload bytes behind on short-buffer paths, which may desynchronize
subsequent reads.
- Stream deadlines were previously applied to the shared underlying session connection, which could affect unrelated
streams multiplexed over the same session.

Suggested changes

1. Strengthen session lifecycle management
- Introduce an explicit session state model (active / idle / closing / closed).
- Track active stream count per session.
- Add idle timeout cleanup for idle sessions.
- Ensure dialer shutdown closes all underlying AnyTLS sessions.

2. Validate idle sessions before reuse
- Before reusing an idle session that has been idle for a while, send a lightweight heartbeat/probe.
- If the probe fails, discard the session instead of reusing it.

3. Expose runtime stats
- Add AnyTLS-specific runtime diagnostics:
- anytlsDialers
- anytlsSessions
- anytlsIdleSessions
- anytlsActiveStreams
- These stats should be available through the existing runtime diagnostics path.

4. Fix stream-layer protocol correctness
- Enforce the uint16 frame payload limit.
- Split oversized PSH payloads into multiple frames instead of writing invalid frame lengths.
- Drain the remaining UDP payload in packetStream.ReadFrom() when returning io.ErrShortBuffer.
- Make stream read deadlines stream-local instead of applying them to the shared underlying connection.

5. Long-term improvement
- The remaining semantic gap is WriteDeadline on multiplexed streams.
- A fully correct WriteDeadline likely requires a dedicated session writer loop with per-stream request scheduling,
rather than directly writing from each stream into the shared connection.

Why this matters

These changes improve long-running stability under router workloads, reduce stale session reuse, improve
debuggability, and fix protocol-level correctness issues without changing the wire format.

### Expected Behavior

_No response_

### Steps to Reproduce

Reproduction Conditions

The issue is most likely to appear under these conditions:

1. daed is using the current AnyTLS path based on github.com/daeuniverse/outbound/protocol/anytls.
2. The active outbound node is an AnyTLS node.
3. The router runs for a long period under real traffic, typically 8-24 hours.
4. There is sustained concurrent TCP activity, especially many short-lived and medium-lived connections.
5. DNS and proxied traffic are both active, so the router continuously opens and closes streams over reused AnyTLS
sessions.
6. Configuration reloads, node switching, or service restarts can make the problem easier to trigger.

How to Reproduce

1. Configure daed to use an AnyTLS outbound node.
2. Route normal client traffic through daed.
3. Generate mixed traffic for several hours, for example:
- web browsing
- video streaming
- repeated HTTPS requests
- chat applications
- background OS update traffic
4. Optionally make the issue easier to trigger by:
- switching nodes
- reloading daed
- changing routing or DNS-related settings
5. Observe the daed process over time.

What to Observe

Monitor these indicators periodically:

- Resident memory:

cat /proc/$(pidof daed)/status | grep VmRSS
- File descriptor count:

ls /proc/$(pidof daed)/fd | wc -l
- Socket states:

netstat -tanp 2>/dev/null | grep $(pidof daed)
- Log patterns:

tail -n 200 /var/log/daed/daed.log
logread | grep -E 'daed|closed network connection|closed pipe|panic|fatal'

Expected Problematic Symptoms

When the issue is reproduced, one or more of the following may appear:

1. VmRSS keeps growing over time instead of stabilizing.
2. File descriptor count keeps increasing over time.
3. Large numbers of sockets accumulate, especially abnormal states such as CLOSE_WAIT in older reproductions.
4. Logs repeatedly show messages such as:
- use of closed network connection
- read/write on closed pipe
5. The growth continues even though traffic volume is not increasing proportionally.

Notes

- This issue is easier to reproduce with real-world long-running traffic than with a short synthetic test.

### Environment

Environment

- Firmware: custom OpenWrt/LEDE build based on this repository
- OpenWrt: 24.10.5
- Revision: R26.02.20
- Target: rockchip/armv8
- Architecture: aarch64_generic
- Kernel version: 6.12.80
- Device hostname: LEDE

- daed version: 1.27.0
- daed binary: /usr/bin/daed
- daed runtime mode: procd-managed service
- daed API/listen address: 0.0.0.0:2023

- Proxy protocol under test: AnyTLS
- Observed remote AnyTLS server during testing: 14.17.78.162:33423

- DNS:
- mosdns enabled
- mosdns config path: /var/etc/mosdns.json
- mosdns local listen port: 5335

- Workload:
- real user traffic from LAN devices
- long-running mixed traffic including DNS, HTTPS, browsing, streaming, and background application traffic

### Anything else?

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in github.com/daeuniverse/outbound/protocol/anytls and inspect packetStream.ReadFrom, the session and dialer lifecycle, and the existing runtime diagnostics path. Reproduce the long-running AnyTLS workload if possible, then verify session cleanup, idle-session health checks, runtime counts, bounded frame handling, payload draining, and stream-local deadlines without changing the wire format.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, networking, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.