gRPC OTLP input never sends ExportTraceServiceResponse over mTLS
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
**Describe the bug**
When the OpenTelemetry input plugin (`in_opentelemetry`) receives a gRPC OTLP trace export
request over a **mTLS connection**, it correctly receives and processes the span, but **never
sends the `ExportTraceServiceResponse` back to the client**. The gRPC call hangs indefinitely
until the client closes the connection (timeout, RST_STREAM, or server-side IO timeout).
With **plaintext gRPC** (no TLS), the response is sent immediately and correctly.
**To Reproduce**
- Configure `[INPUT] opentelemetry` on port 4317 with mTLS enabled
- Send a gRPC OTLP trace export request from a Java gRPC client (e.g. using `TraceServiceGrpc.newFutureStub`)
- Observe that fluent-bit receives and stores the span but the gRPC client never receives `ExportTraceServiceResponse`
- The client only unblocks when the server's IO timeout fires (e.g. if `net.io_timeout` configured in the INPUT) or the client closes the connection
**Expected behavior**
Fluent-bit sends `ExportTraceServiceResponse` immediately after processing the span, regardless of whether TLS is used.
**Your Environment**
- Fluent Bit version: 4.2.3
- Transport: gRPC over mTLS (SPIRE / SVID certificates)
- Input plugin: `in_opentelemetry`, port 4317
**Additional context**
## Root Cause Analysis
The bug is a **coroutine context mismatch** in the HTTP server's TLS write path. It is related
to #11551 but affects a different code path (HTTP/2 server response write, not the forward
input read loop).
### Call chain when sending the gRPC response
```
flb_http_server_client_activity_event_handler() ← FLB_ENGINE_EV_CUSTOM (non-coroutine)
→ flb_http_server_session_write()
→ flb_io_net_write()
→ flb_tls_net_write_async(co, ...) ← co == NULL (not in a coroutine context)
```
### The stall
Under mTLS, OpenSSL returns `SSL_ERROR_WANT_READ` during the write (post-handshake
authentication). `flb_tls_net_write_async` handles this as:
```c
if (ret == FLB_TLS_WANT_READ) {
io_tls_event_switch(session, MK_EVENT_READ); // re-registers fd as FLB_ENGINE_EV_THREAD
flb_coro_yield(co, FLB_FALSE); // co == NULL → undefined behaviour
goto retry_write;
}
```
Two problems occur simultaneously:
1. `io_tls_event_switch` re-registers the fd as `FLB_ENGINE_EV_THREAD`, **detaching it from
the HTTP server's `FLB_ENGINE_EV_CUSTOM` handler**. The HTTP server loses ownership of
the fd.
2. `flb_coro_yield(NULL, ...)` is called with a NULL coroutine — there is no coroutine to
resume when the read event fires.
When the socket becomes readable, the engine dispatches to the coroutine scheduler which
attempts to resume a NULL coroutine — a no-op. The response data remains stuck in the
`outgoing_data` buffer forever and the fd is orphaned.
### Why the span is stored despite the missing ACK
The span **is** stored correctly by fluent-bit (the `[INPUT]` side works). Only the gRPC ACK
(`ExportTraceServiceResponse`) is never sent.
When the gRPC client's deadline fires and sends `RST_STREAM`, the resulting fd error triggers
session teardown which flushes kernel-level TCP buffers — but by then it is too late for a
proper gRPC response.
### nghttp2 send callback is not a socket write
For additional context: `nghttp2`'s send callback (`http2_send_callback`) does **not** write
to the socket — it only appends serialized HTTP/2 frames to an in-memory `outgoing_data` SDS
buffer. The actual socket write happens later via `flb_http_server_session_write()`, which is
the point where the TLS stall occurs.
## Suggested fix directions
1. **Null-check on `co` in `flb_tls_net_write_async`** — fall back to synchronous
`flb_tls_net_write` when not in a coroutine context, avoiding the yield on a NULL coroutine.
2. **Run the HTTP server session write handler inside a coroutine** — so `flb_coro_get()`
returns a valid context when `flb_io_net_write` is called.
3. **Separate response flush from the read event** — after building the response, register the
fd for `MK_EVENT_WRITE` and flush `outgoing_data` from a dedicated write event, avoiding
the mixed read/write event ownership problem entirely.
Contributor guide
Assessment
This issue has not been assessed yet.