`nerdctl run` with attached stdio stops draining container stdout at ~72 KiB, deadlocking the container (nerdctl 2.3.5, containerd 2.3.3, Kata runtime-rs)

## Summary

With `nerdctl run --rm -i` (foreground, non-TTY, stdio attached), the attach-side stdout consumer stops reading after roughly 72 KiB have been delivered to the host. The container's writer then blocks forever on a full pipe and the whole chain deadlocks: the container never exits, `nerdctl` never returns, and signals sent to the `nerdctl` client do not tear the task down.

The same image, same containerd, same runtime, driven through containerd's own `ctr run` client instead of `nerdctl`, streams 64 MiB of stdout cleanly. That is what points at the nerdctl attach path rather than at the runtime or the guest.

## Environment

| Component | Version / detail |
|---|---|
| nerdctl | 2.3.5 (official release binary, checksum-verified) |
| containerd | 2.3.3 |
| Runtime | Kata Containers 4.0.0, **runtime-rs** shim (`containerd-shim-kata-v2` from the kata-static `runtime-rs/bin` tree) |
| Hypervisor actually used | QEMU (`qemu-system-x86_64`), via the shim's vendored `configuration-qemu-runtime-rs.toml` |
| Guest stdio pipe size | Kata `container_pipe_size` effective value `0` → OS default (~64 KiB) |
| Host OS | Ubuntu 24.04.4, kernel 6.17.0-1021-azure (x86_64, KVM) |
| Guest kernel | 6.18.35 |
| runc | **Not installed on this host — see "What was and was not tested"** |
| BuildKit / CNI | not installed; container had `--network none` |

## Reproduction

Everything below is generic; run as root on a host with containerd + nerdctl + a Kata runtime handler registered. `$RUNTIME` is the Kata shim handler name (ours resolved to the runtime-rs shim); `$IMAGE` is any image with a POSIX shell and `dd` already present in the content store (we used `--pull never` on an air-gapped host).

**A. Through nerdctl — stalls**

```sh
# 4 MiB of stdin, consumed to EOF first, then 8 MiB written to stdout
head -c 4194304 /dev/zero > /tmp/stdin.bin

timeout 90 nerdctl run --rm -i --pull never --runtime "$RUNTIME" "$IMAGE" \
sh -c 'cat > /dev/null; dd if=/dev/zero bs=1M count=8 2>/dev/null' \
< /tmp/stdin.bin > /tmp/out.bin
echo "exit=$?"
wc -c < /tmp/out.bin
```

Observed: `nerdctl` never returns; `timeout` fires (`exit=124`). Sampling `/tmp/out.bin` while the run is wedged shows delivery starting and then stopping — in our live-sampled reproduction it stopped at exactly **73,728 bytes** (72 KiB = a 64 KiB pipe plus an 8 KiB copy buffer) and never advanced again.

**B. Same image and runtime through `ctr` — completes**

```sh
timeout 90 ctr run --rm --runtime "$RUNTIME" "$IMAGE" probe-ctr \
sh -c 'dd if=/dev/zero bs=1M count=8 2>/dev/null' > /tmp/out2.bin
echo "exit=$?"
wc -c < /tmp/out2.bin
```

Observed: `exit=0`, all 8,388,608 bytes delivered. We ran this control across a size sweep of 0.25 / 0.5 / 1 / 2 / 4 / 8 / 16 / 32 / 64 MiB, with both a single monolithic `write()` and 64 KiB chunked writes awaiting drain, and with a bidirectional shape (4 MiB in to EOF, then 30 MiB out). Every cell passed, including 64 MiB monolithic (all bytes delivered, ~8.7 s wall including VM boot) and the 30 MiB bidirectional case (~1.2 s in-guest).

## Observed vs expected

**Expected:** `nerdctl run` with attached stdio keeps draining the container's stdout for as long as the container writes, as `ctr` does; the container runs to completion and `nerdctl` exits with the container's status.

**Observed:**

1. Host-side stdout delivery stops at ~72 KiB (73,728 bytes in the sampled run) and never resumes.
2. The guest writer blocks on the now-full pipe; the container never exits and CPU goes idle.
3. Thread/wchan sampling of the wedged process chain shows the **Kata runtime-rs shim's tokio worker blocked in `anon_pipe_write`** — i.e. it is trying to write stdout *toward nerdctl* — while **every `nerdctl` thread sits idle in `futex`/`epoll`**. Nothing on the nerdctl side is consuming.
4. Signals do not get through the wedged attach: `SIGTERM` to the `nerdctl` client (via `timeout`) killed nothing; one specimen sat wedged for 19+ minutes.
5. Recovery through nerdctl is also blocked: `nerdctl rm --force ` hangs. Killing the `nerdctl` client leaves the shim, the hypervisor process and the virtiofsd processes alive (a leaked VM). `ctr task kill -s SIGKILL ` returns 0 but the task stays `RUNNING`; `ctr task rm -f` hangs. Only killing the hypervisor and virtiofsd PIDs first, then the shim, reclaims everything — after which `ctr container rm` and `nerdctl rm --force` both succeed.

Point 3 is the core of the report: the writer is blocked writing to nerdctl, and nerdctl is asleep. That is a stalled reader on the attach side rather than a runtime or guest-agent problem.

## The ~72 KiB threshold

73,728 bytes = 64 KiB + 8 KiB, which lines up with a default-sized (64 KiB) stdio pipe plus one 8 KiB copy buffer already consumed. That is consistent with the attach-side reader performing one buffer's worth of copying and then never scheduling another read, leaving the pipe permanently full. We did not bisect the threshold with a size sweep through nerdctl, so treat 73,728 as one precisely measured wedge point rather than a proven constant; the qualitative finding is that small outputs (diagnostics, short JSON) always got through and every real payload never did.

## What was and was not tested

Stated explicitly so nothing here is over-claimed:

- **Reproduced:** nerdctl 2.3.5 + containerd 2.3.3 + Kata 4.0.0 runtime-rs shim under QEMU, non-TTY, `--interactive`, with stdin fed and read to EOF before the large stdout write. Reproduced with a real workload emitting a large JSON result, and confirmed by live thread/wchan sampling of the wedged chain.
- **Ruled out (same host, same runtime, same image):** the Kata guest, the guest agent, and the runtime-rs shim's own stdio copy path — `ctr` drains the identical FIFOs at up to 64 MiB without issue, monolithic and chunked alike. Request/stdin size was never a factor (a 35 KB stdin request reproduced the stall).
- **NOT tested: runc.** `runc` is deliberately not installed on this host (the only registered runtime is the Kata shim), so we have **no** data on whether the same stall occurs with `nerdctl run --runtime runc`. We therefore cannot say whether this is Kata-specific or a general property of nerdctl's attach path. The evidence isolates the fault to nerdctl's attach-side reader *relative to `ctr` on the same runtime*, and no further.
- **NOT tested:** TTY mode (`-t`), detached mode with `nerdctl logs`, running without `--interactive`/without stdin attached, other log drivers, and other nerdctl versions. Every nerdctl reproduction we have involves `-i` with stdin attached, so we cannot say whether stdin attachment is required to trigger it.
- **NOT tested:** whether raising Kata's `container_pipe_size` (which is honored end-to-end down to `F_SETPIPE_SZ` in the guest, but is `0`/default in the shipped QEMU runtime-rs template) merely raises the threshold. It would not address a reader that stops reading.

## Impact / workaround

Any workload whose stdout exceeds ~72 KiB under `nerdctl run` with attached stdio deadlocks permanently and leaks the sandbox VM, since neither signal proxying nor `nerdctl rm --force` can reach it. Our workaround was to move all bulk output off stdio entirely (onto a bind-mounted output file), keeping stdio for small control-plane messages only, plus an out-of-band PID-kill escalation for teardown. Happy to run further probes on this host if a maintainer wants a specific variant tested (TTY, no-stdin, alternate log driver).