microsoft / microsoft/Windows-Containers

Process-isolated container remains "Running" for exactly TcpTimedWaitDelay after all processes exit, if a loopback TCP connection was closed

Open
#644 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug triage
Dominant language
PowerShell
Stars
551
Forks
76
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

A process-isolated Windows container whose workload opened and closed a loopback (127.0.0.1) TCP connection is not reported as stopped when its last process exits. Instead, the compute system remains in Running state for exactly the host's effective TcpTimedWaitDelay (default 120 seconds), measured from the moment the last loopback connection was closed. During this window no container processes exist (verified on the host with System Informer / Process Explorer), docker ps still shows the container as Running, docker events emits the die event only at the end of the window, and docker exec into the container hangs and eventually fails with container ... is not running.

The closed connection's TIME_WAIT TCBs are kernel-owned (not attributable to any process) and appear to pin the container's network compartment, which blocks silo/compute-system teardown until they expire. The delay scales linearly with TcpTimedWaitDelay (120 → ~120 s, 30 → ~30 s), which confirms the mechanism. Hyper-V-isolated containers are unaffected. The container engine version is irrelevant (reproduced identically on Docker Engine 25.0.4 and 29.5.2); the wait happens below the engine, in HCS.

Practical impact: any container workload that talks to a localhost helper daemon — compiler caches like sccache, local package proxies, etc. — silently adds up to 2 minutes to every container run. In our CI, every Windows build job carried an invisible, exactly-120-second gap at the end of the build step. Nothing in user space is observable during the window, which makes this extremely hard to diagnose.

To Reproduce

Steps to reproduce the behavior:

  1. On a Windows 11 25H2 host (build 10.0.26200.8875) with Docker and process isolation, run a container whose entrypoint opens and closes a single loopback TCP connection and then exits:
Measure-Command { docker run --rm --isolation=process mcr.microsoft.com/windows/servercore:ltsc2022 powershell -Command '$l=[Net.Sockets.TcpListener]::new([Net.IPAddress]::Loopback,0); $l.Start(); $c=[Net.Sockets.TcpClient]::new(); $c.Connect("127.0.0.1",$l.LocalEndpoint.Port); $s=$l.AcceptTcpClient(); $c.Close(); $s.Close(); $l.Stop(); "done"' }
  1. Run the identical control container with no TCP activity:
Measure-Command { docker run --rm --isolation=process mcr.microsoft.com/windows/servercore:ltsc2022 powershell -Command '"done"' }
  1. Compare wall-clock times. Optionally observe during the gap: docker ps (container Running), process list on the host (no container processes), docker events (die delayed), docker exec (hangs, then container is not running).

Measured results across two hosts on the same OS build:

Host Engine TcpTimedWaitDelay Isolation Repro (1 TCP conn) Control (no TCP)
A Docker 25.0.4 not set (default 120 s) process 128.7 s 10.7 s
B Docker 29.5.2 30 (0x1e) process 32.9 s 4.2 s
B Docker 29.5.2 30 hyperv 5.1 s 4.9 s

(StrictTimeWaitSeqCheck is not set on either host.)

Expected behavior

The container is reported as stopped promptly after its last process exits. TIME_WAIT drain of already-closed sockets should not gate compute-system teardown — for comparison, the equivalent Linux container reports its exit immediately (a lingering network namespace does not delay the container's exit status).

Configuration:

  • Edition: Windows 11 Pro 25H2, build 10.0.26200.8875 (two independent hosts)
  • Base Image being used: mcr.microsoft.com/windows/servercore:ltsc2022 (cross-version process isolation on a 26200 host; also reproduced with a custom image derived from it)
  • Container engine: docker (Docker Engine, Windows containers)
  • Container Engine version: 25.0.4 and 29.5.2 (identical behavior on both)

Additional context

  • The delay is anchored at the close of the last loopback TCP connection, not at process exit — if the connection closes earlier in the container's life, the visible tail delay shrinks accordingly.
  • Workaround: setting HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpTimedWaitDelay to 30 (the minimum) on the host shortens the delay to ~30 s, but cannot eliminate it.
  • Real-world trigger: sccache (a compiler cache) runs a localhost TCP server inside the container; its final sccache --stop-server call closes the last connection right at the end of the CI script, so every CI build job paid the full 120 s.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the two PowerShell docker run measurements from the reproduction steps, comparing process and Hyper-V isolation and varying TcpTimedWaitDelay. Start by tracing the Windows HCS compute-system teardown path for process-isolated containers and its handling of kernel-owned TIME_WAIT connections; done means the container reports stopped promptly after its last process exits without waiting for TCP timeout expiry.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, powershell
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.