fluent / fluent/fluent-bit

Synchronous socket writes add one-second delays after EAGAIN

Open
#12,115 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

**Describe the bug**

The synchronous socket writer in `fd_io_write()` retries a nonblocking
`send()` or `sendto()` after `EAGAIN`/`EWOULDBLOCK` by sleeping for one full
second.

When a large response fills the socket buffer repeatedly, each temporary
backpressure event adds another second to response completion even if the
socket becomes writable much sooner.

The behavior is present in Fluent Bit 5.0.2 and current `master`.

**To Reproduce**

1. Create a nonblocking socket pair.
2. Set the writer's send buffer to 4096 bytes.
3. Write a 16 KiB payload through `flb_io_fd_write()`.
4. Start a reader thread that waits 20 ms before draining the peer.
5. Measure the synchronous write duration.

A focused internal regression test reproduces the behavior. With the current
implementation, the one-second retries make the test exceed its 900 ms bound.
After replacing the blind sleep with a writability wait, the same write
completes in approximately 20-30 ms.

**Expected behavior**

After `EAGAIN`/`EWOULDBLOCK`, the synchronous writer should wait for `POLLOUT`
with the existing one-second upper bound and retry as soon as the socket is
writable. The existing bounded retry behavior should remain in place.

**Your Environment**

- Version used: 5.0.2 and master at `fd5ea1f0d503`
- Operating systems: embedded Linux arm64 and Debian 12 x86_64 test container
- Affected path: synchronous network writes in `src/flb_io.c`
- Observed with: a large `prometheus_exporter` metrics response

**Additional context**

The issue was isolated while testing a large metrics response over a direct
local network connection.

Before the fix, complete responses of approximately 2.69 MiB took 16-17
seconds. A syscall trace correlated the delay with sixteen `EAGAIN` results and
sixteen one-second sleeps in the HTTP worker.

With a bounded `POLLOUT` wait, a traced 2,716,787-byte response completed in
0.252 seconds. The worker encountered 23 `EAGAIN` results, performed 23
writability waits, and performed zero one-second sleeps.

The proposed regression test fails with the old implementation and passes with
the writability wait.

Contributor guide

Open the contributing guide

Research direction

Start in src/flb_io.c at fd_io_write() and the flb_io_fd_write() path, then run the focused internal regression test described in the report. Reproduce the nonblocking socket-pair case with a 16 KiB payload and delayed reader. Done means EAGAIN/EWOULDBLOCK waits for POLLOUT within the existing one-second bound, retries promptly, and the regression test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.