out_gelf: UDP socket is never recreated after send() fails, all subsequent records are lost
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
**Describe the bug**
`out_gelf` in UDP mode creates its socket once in `cb_gelf_init()` and never recreates it. If that socket becomes unusable at runtime, every subsequent `send()` fails with the same error forever and all records are lost — the plugin keeps writing to the same broken fd until the process is restarted.
To be clear about what is and isn't being reported: a datagram lost in the network is expected for UDP and is not a bug. Here the process gets a synchronous, deterministic answer from its own kernel saying "this socket can no longer send", and discards that information.
All line references are permalinks to tag `v4.2.2` ([`ddfef36`](https://github.com/fluent/fluent-bit/commit/ddfef360d7f3ac5268942c47ccc9b01864424a05)), the version we run.
The socket is created once, in the init callback: https://github.com/fluent/fluent-bit/blob/ddfef360d7f3ac5268942c47ccc9b01864424a05/plugins/out_gelf/gelf.c#L478-L492
There are three `send()` sites on the UDP path and none of them touches the socket:
* [`gelf_send_udp_pckt()`, L189-L196](https://github.com/fluent/fluent-bit/blob/ddfef360d7f3ac5268942c47ccc9b01864424a05/plugins/out_gelf/gelf.c#L189-L196) — the non-chunked branch calls `flb_errno()` and returns `-1`, which does reach the engine as a retry.
* [`gelf_send_udp_chunked()`, L170-L179](https://github.com/fluent/fluent-bit/blob/ddfef360d7f3ac5268942c47ccc9b01864424a05/plugins/out_gelf/gelf.c#L170-L179) — logs via `flb_errno()` but has no `return` in the error path and ends with `return 0`, so it reports success even when every chunk failed. Its caller ([L187-L189](https://github.com/fluent/fluent-bit/blob/ddfef360d7f3ac5268942c47ccc9b01864424a05/plugins/out_gelf/gelf.c#L187-L189)) discards the return value anyway. With the defaults `compress true` and `packet_size 1420`, any message that doesn't fit in one packet takes this path, so a fully failed send is reported to the engine as delivered — no retry, no dropped-records metric.
* [`gelf_send_udp()`, L221-L226](https://github.com/fluent/fluent-bit/blob/ddfef360d7f3ac5268942c47ccc9b01864424a05/plugins/out_gelf/gelf.c#L221-L226) — returns the error without logging it.
`close(ctx->fd)` appears only in the init error path and in [`cb_gelf_exit()`](https://github.com/fluent/fluent-bit/blob/ddfef360d7f3ac5268942c47ccc9b01864424a05/plugins/out_gelf/gelf.c#L528-L532). Nothing reconnects or recreates the socket after a send error, so `FLB_RETRY` just repeats `send()` on the same dead fd.
**To Reproduce**
We hit this on Kubernetes with Cilium socket load balancing, but the root cause is generic: any event that leaves a connected UDP socket disconnected reproduces it.
1. `out_gelf` with `Mode udp` pointing at a Kubernetes ClusterIP service.
2. Cilium with kube-proxy replacement and socketLB rewrites the service IP to a backend pod IP inside `connect()`.
3. Delete that backend pod. Cilium terminates sockets pinned to it (cilium/cilium#25169, extended to pod netns in cilium/cilium#33459). For UDP the kernel path is `udp_abort()` → `__udp_disconnect()`, which **disconnects** the socket instead of closing the fd.
4. The socket is now unconnected, so `send()` — which carries no destination address — fails permanently:
```
[error] [/src/fluent-bit/plugins/out_gelf/gelf.c:193 errno=89] Destination address required
[error] [engine] chunk '1-1789021941.897344143.flb' cannot be retried: task_id=6, input=tail.0 > output=gelf.0
```
`errno=89` is `EDESTADDRREQ`. In our case: 3800 such errors and 1885 dropped chunks in 40 minutes on a single node, and delivery never recovered on its own — only a DaemonSet restart fixed it, which matches the code above, since a new process means a new `cb_gelf_init()`. The real loss is likely higher, because messages taking the chunked path fail without being counted at all.
Note that this is specific to disconnection. An ordinary ICMP port-unreachable sets `sk_err` but leaves `sk_state` and the destination intact, so the next `send()` returns the pending `ECONNREFUSED` once and then recovers by itself. That case self-heals; this one does not.
**Expected behavior**
The UDP path should recreate the socket before retrying when `send()` reports that the socket itself is no longer usable.
A fix already exists in the codebase and doesn't need to be invented: `out_syslog` moved its UDP mode from a raw fd to `flb_upstream` with `FLB_IO_UDP` in #11728. With that, `flb_upstream_conn_get()` checks `flb_socket_error()` (`SO_ERROR`) before handing a pooled connection back, and `flb_io_net_connect()` re-runs `flb_net_udp_connect()` for a new one — so this class of failure is handled generically. The TCP/TLS path of `out_gelf` itself already works this way ([L286-L292](https://github.com/fluent/fluent-bit/blob/ddfef360d7f3ac5268942c47ccc9b01864424a05/plugins/out_gelf/gelf.c#L286-L292)); only the UDP path holds a permanent raw fd.
**Your Environment**
* Version used: 4.2.2. The UDP path is unchanged on current master ([`f784724`](https://github.com/fluent/fluent-bit/commit/f78472443d32691c319c424c75d9ea2343dbc8f7), 2026-09-07): `plugins/out_gelf/gelf.c` is byte-identical to v4.2.2 except for the copyright year. The last commit touching the file is [`a981f66`](https://github.com/fluent/fluent-bit/commit/a981f6681c4b) "plugins: update copyright year to 2026" (2026-01-01); the last functional change is [`876d6e8`](https://github.com/fluent/fluent-bit/commit/876d6e854c0c) (2025-03-17), unrelated to the send path.
* Configuration: `[OUTPUT] Name gelf / Mode udp / Compress true / Host / Port 12201`, with `storage.type filesystem` on the input
* Environment name and version: Kubernetes 1.28.3, Cilium 1.16.19 (kube-proxy replacement, socketLB)
* Operating System and version: Ubuntu, kernel 6.8.0-138
* Filters and plugins: `tail` input, `kubernetes` filter, `out_gelf` output
**Additional context**
The same "connect once in init, never reconnect" pattern is in the other two users of `flb_net_udp_connect()` — `out_udp` (`plugins/out_udp/udp_conf.c`) and `out_syslog` before #11728 — so this looks like a shared pattern rather than something specific to GELF. Both of those at least return the error to the engine instead of reporting success.
`storage.keep_rejected` preserves the rejected chunks on disk, but it doesn't restore delivery, so it isn't a workaround for this.
Logs lost this way are unrecoverable for us: the pods producing them are deleted once their job finishes, so the log server is the only archive. The failure is also invisible from the outside — the server stays healthy and simply receives nothing, which makes it easy to misdiagnose as a server-side problem.
Contributor guide
Research direction
Start in plugins/out_gelf/gelf.c at cb_gelf_init(), gelf_send_udp_pckt(), gelf_send_udp_chunked(), and gelf_send_udp(), then compare the UDP handling with out_syslog's flb_upstream migration in #11728. Check the out_udp pattern as well. Done means an unusable UDP socket is replaced before retry, and chunked send failures are reported as failures rather than success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, kubernetes, linux
- Domain
- devops, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100