containers / containers/conmon

conn_sock.c: 'Failed to write container stdin' is logged on every EAGAIN, flooding the journal for slow stdin consumers

Open Beginner friendly
#689 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
499
Forks
150
Avg merge
1d 10h
Merged PRs (30d)
20

Description

## Summary

`sock_try_write_to_local_sock()` logs `Failed to write container stdin` on **every** `EAGAIN` from the container's non-blocking stdin fd. For any container whose stdin consumer is slower than the writer, this floods the journal at a rate proportional to the bytes fed, while being entirely benign: the same code path retries the unwritten bytes, so nothing is lost.

Measured on a log-analysis pipeline (`podman run --rm -i ... goaccess ... -`, one normalized stream on stdin): **~11,000 journal lines a day** from this one message, all of it inside the systemd unit's own journal. It made `journalctl -u ` unusable exactly when the pipeline needed diagnosing, and the warnings read as input loss on a live rebuild until byte accounting ruled that out.

## Mechanism

Container stdin is set non-blocking (`conmon.c`, `g_unix_set_fd_nonblocking(mainfd_stdin, TRUE, NULL)`; also `conn_sock.c` for the destination fd). In `conn_sock.c`, v2.1.12:

```c
if (local_sock->is_stream) {
w = write(*(local_sock->fd), sock->buf + sock->off, sock->remaining);
} else {
w = sendto(...);
}
if (w < 0) {
nwarnf("Failed to write %s", local_sock->label);
} else {
sock->off += w;
sock->remaining -= w;
}
```

Two things follow:

1. **`errno` is never examined**, so `EAGAIN` (the pipe is simply full because the consumer is busy) logs identically to a real failure.
2. **`off`/`remaining` are left untouched on error**, so `write_to_local_sock()` sets `has_data`, `local_sock_write_cb()` returns `G_SOURCE_CONTINUE`, and the same bytes are written when the fd is next writable. The warning therefore reports a *successful retry loop*, one line per retry.

`main` still warns unconditionally; it only changed the message to `pwarnf("Failed to write to fd %s", ...)`, which usefully appends `strerror(errno)` but does not make the warning conditional.

## Reproduction

Two arms, identical except podman's log level, feeding ~40 MB of 200-byte lines into a container whose reader is a shell `read` loop (slow, line-oriented, so the pipe stays at capacity):

```
yes "$(head -c 200 /dev/zero | tr '\0' 'x')" | head -n 200000 \
| podman [--log-level=error] run --rm -i --network none \
--name arm --entrypoint sh -c 'while IFS= read -r l; do :; done'
```

Warnings attributed by container id from `journalctl -t conmon`:

| conmon `--log-level` | `Failed to write container stdin` |
|---|---|
| `warning` (podman default) | **2,323** |
| `error` | **0** |

Versions: podman 5.4.2, conmon 2.1.12, crun 1.21, Debian packages.

## Suggested fix

Skip the warning when the write failed only because the fd was not ready:

```c
if (w < 0 && errno != EAGAIN && errno != EWOULDBLOCK)
pwarnf("Failed to write to fd %s", local_sock->label);
```

`EAGAIN` on a non-blocking fd whose bytes are retried is a normal event in this loop, not a warning. Demoting it to `ndebugf` would work equally well for anyone who wants to see it.

## Workaround

Lowering podman's log level for the affected invocation (`podman --log-level=error run ...`) propagates to conmon and silences the class, at the cost of every other podman/conmon warning for that run. That is what we shipped, but it is a mitigation rather than a fix.

Related: #576 notes `conn_sock.c` could use an overhaul generally; this is one specific, self-contained symptom.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read conn_sock.c, starting at sock_try_write_to_local_sock() and the local_sock_write_cb() retry path. Reproduce with the provided slow stdin pipeline and inspect conmon journal output. Done means benign EAGAIN retries no longer flood warnings while genuine write failures remain visible.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
devops, tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.