containers / containers/conmon
conn_sock.c: 'Failed to write container stdin' is logged on every EAGAIN, flooding the journal for slow stdin consumers
- Dominant language
- C
- Stars
- 499
- Forks
- 150
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 20
Description
## Summary
`sock_try_write_to_local_sock()` logs `Failed to write container stdin` on **every** `EAGAIN` from the container's non-blocking stdin fd. For any container whose stdin consumer is slower than the writer, this floods the journal at a rate proportional to the bytes fed, while being entirely benign: the same code path retries the unwritten bytes, so nothing is lost.
Measured on a log-analysis pipeline (`podman run --rm -i ... goaccess ... -`, one normalized stream on stdin): **~11,000 journal lines a day** from this one message, all of it inside the systemd unit's own journal. It made `journalctl -u ` unusable exactly when the pipeline needed diagnosing, and the warnings read as input loss on a live rebuild until byte accounting ruled that out.
## Mechanism
Container stdin is set non-blocking (`conmon.c`, `g_unix_set_fd_nonblocking(mainfd_stdin, TRUE, NULL)`; also `conn_sock.c` for the destination fd). In `conn_sock.c`, v2.1.12:
```c
if (local_sock->is_stream) {
w = write(*(local_sock->fd), sock->buf + sock->off, sock->remaining);
} else {
w = sendto(...);
}
if (w < 0) {
nwarnf("Failed to write %s", local_sock->label);
} else {
sock->off += w;
sock->remaining -= w;
}
```
Two things follow:
1. **`errno` is never examined**, so `EAGAIN` (the pipe is simply full because the consumer is busy) logs identically to a real failure.
2. **`off`/`remaining` are left untouched on error**, so `write_to_local_sock()` sets `has_data`, `local_sock_write_cb()` returns `G_SOURCE_CONTINUE`, and the same bytes are written when the fd is next writable. The warning therefore reports a *successful retry loop*, one line per retry.
`main` still warns unconditionally; it only changed the message to `pwarnf("Failed to write to fd %s", ...)`, which usefully appends `strerror(errno)` but does not make the warning conditional.
## Reproduction
Two arms, identical except podman's log level, feeding ~40 MB of 200-byte lines into a container whose reader is a shell `read` loop (slow, line-oriented, so the pipe stays at capacity):
```
yes "$(head -c 200 /dev/zero | tr '\0' 'x')" | head -n 200000 \
| podman [--log-level=error] run --rm -i --network none \
--name arm --entrypoint sh -c 'while IFS= read -r l; do :; done'
```
Warnings attributed by container id from `journalctl -t conmon`:
| conmon `--log-level` | `Failed to write container stdin` |
|---|---|
| `warning` (podman default) | **2,323** |
| `error` | **0** |
Versions: podman 5.4.2, conmon 2.1.12, crun 1.21, Debian packages.
## Suggested fix
Skip the warning when the write failed only because the fd was not ready:
```c
if (w < 0 && errno != EAGAIN && errno != EWOULDBLOCK)
pwarnf("Failed to write to fd %s", local_sock->label);
```
`EAGAIN` on a non-blocking fd whose bytes are retried is a normal event in this loop, not a warning. Demoting it to `ndebugf` would work equally well for anyone who wants to see it.
## Workaround
Lowering podman's log level for the affected invocation (`podman --log-level=error run ...`) propagates to conmon and silences the class, at the cost of every other podman/conmon warning for that run. That is what we shipped, but it is a mitigation rather than a fix.
Related: #576 notes `conn_sock.c` could use an overhaul generally; this is one specific, self-contained symptom.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read conn_sock.c, starting at sock_try_write_to_local_sock() and the local_sock_write_cb() retry path. Reproduce with the provided slow stdin pipeline and inspect conmon journal output. Done means benign EAGAIN retries no longer flood warnings while genuine write failures remain visible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- devops, tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100