MirrorNetworking / MirrorNetworking/Mirror

Bug Report: Unbatcher pooled writers leak on disconnect paths (UnpackAndInvoke failure / Cleanup)

Open
#4,125 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.3k
Forks
870
PR merge metrics
No merged PRs in 30d

Description

Environment

  • Mirror version: 96.11.0 (upgraded from 96.6.5; leak reproduced on both)
  • Unity: 6000.5, IL2CPP, Linux dedicated server (headless, Development Build)
  • Transport: kcp2k (MultiplexTransport wrapper), ReliableMaxMessageSize = 297433, SendWindowSize = 256
  • Scale: up to ~70 concurrent players, ~10k+ connects/disconnects per day

Symptom

Managed heap grows linearly at ~17 KB/s (~1.5 GB per 24h) with zero Unity-object growth. Unity Memory Profiler snapshots taken 4 hours apart (dedicated server, Development Build) show:

Type Count diff (4h) Size diff (4h)
System.Byte[] +175,055 +351.3 MB (single objects of exactly 290.5 KB = array length 297,434)
Mirror.NetworkWriterPooled +155,267 (0 → 155,267) +5.9 MB
System.Text.UTF8Encoding +162,284 (50 → 162,334) +9.9 MB

The three counts line up 1:1:1 — each leaked NetworkWriterPooled holds one 290 KB buffer (expanded to batch size) and one UTF8Encoding. The byte[] reference count is 1 (held by the writer). GC.Collect() only reclaims ~4% — the writers are strongly referenced, not garbage.

Root cause

NetworkConnection.Cleanup() only returns send-side Batcher writers to the pool. The receive-side Unbatcher queue is never cleared on disconnect:

// NetworkConnection.cs (v96.11.0)
public virtual void Cleanup()
{
    foreach (Batcher batcher in batches.Values)
    {
        batcher.Clear();
    }
    // <-- unbatcher on NetworkConnectionToClient is never cleared
}

Leak paths (all present in v96.11.0):

  1. UnpackAndInvoke failure (NetworkServer.cs, OnTransportData): on unknown message id / invalid header the server calls connection.Disconnect() and returns — the unbatcher queue still holds the unread batch writer. #4115 added Unbatcher.Clear() only for the malformed-batch-throw paths (size prefix overflow / size > remaining), not for this unpack-failure path.
  2. Cleanup() on any disconnect: any disconnect with a non-empty unbatcher queue (weak-network race, mid-batch drop) leaks the queued writers.
  3. Client side: NetworkClient.Initialize() replaces unbatcher = new Unbatcher() without returning the old queue's writers to the pool (host-mode reconnect / scene transitions).

Trigger (why it explodes in production)

Mixed client versions: legacy clients send messages the server can't parse → Unknown message id → unpack fails → disconnect (with exceptionsDisconnect = true) → client auto-reconnects → repeats at a mechanical ~10 cycles/second. Each cycle leaks 1 writer whose buffer was expanded to the batch size (legacy clients send large 290 KB batches). 4 hours = 155K leaked writers / 351 MB.

With perfectly synchronized server+client versions this path is rarely hit (batches are normally drained before disconnect), which is why the leak stays silent in official test scenarios — but it is a real disconnect-path defect regardless.

Suggested fix (complements #4115)

  1. In NetworkServer.OnTransportData, clear before disconnecting on unpack failure:
    if (!UnpackAndInvoke(connection, reader, channelId))
    {
        // clear unread batches before disconnecting to prevent writer leaks
        connection.unbatcher.Clear();
        ...
        connection.Disconnect();
    }
    
  2. Override cleanup on the server connection:
    // NetworkConnectionToClient
    public override void Cleanup()
    {
        base.Cleanup();
        unbatcher.Clear();
    }
    
  3. In NetworkClient.Initialize(), call unbatcher.Clear() before unbatcher = new Unbatcher(); (client-side symmetry).

Verification

After applying all three locally, the same 4-hour capture on the same server shows zero NetworkWriterPooled / UTF8Encoding / 290 KB byte[] growth. No behavior change on the normal batch read/write paths (GetNextMessage logic untouched).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with NetworkConnection.Cleanup, NetworkServer.OnTransportData and NetworkClient.Initialize, tracing how Unbatcher queues retain pooled writers on disconnect and replacement. Apply the three proposed cleanup points, then compare the same four-hour capture for NetworkWriterPooled, UTF8Encoding and 290 KB byte[] growth while confirming normal batch read/write behavior is unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, unity
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.