fluent / fluent/fluent-bit

gRPC OTLP input never sends ExportTraceServiceResponse over mTLS

Open
#11,629 3 comments 1 reaction 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

**Describe the bug**
When the OpenTelemetry input plugin (`in_opentelemetry`) receives a gRPC OTLP trace export
request over a **mTLS connection**, it correctly receives and processes the span, but **never
sends the `ExportTraceServiceResponse` back to the client**. The gRPC call hangs indefinitely
until the client closes the connection (timeout, RST_STREAM, or server-side IO timeout).

With **plaintext gRPC** (no TLS), the response is sent immediately and correctly.

**To Reproduce**

- Configure `[INPUT] opentelemetry` on port 4317 with mTLS enabled
- Send a gRPC OTLP trace export request from a Java gRPC client (e.g. using `TraceServiceGrpc.newFutureStub`)
- Observe that fluent-bit receives and stores the span but the gRPC client never receives `ExportTraceServiceResponse`
- The client only unblocks when the server's IO timeout fires (e.g. if `net.io_timeout` configured in the INPUT) or the client closes the connection

**Expected behavior**
Fluent-bit sends `ExportTraceServiceResponse` immediately after processing the span, regardless of whether TLS is used.

**Your Environment**
- Fluent Bit version: 4.2.3
- Transport: gRPC over mTLS (SPIRE / SVID certificates)
- Input plugin: `in_opentelemetry`, port 4317

**Additional context**

## Root Cause Analysis

The bug is a **coroutine context mismatch** in the HTTP server's TLS write path. It is related
to #11551 but affects a different code path (HTTP/2 server response write, not the forward
input read loop).

### Call chain when sending the gRPC response

```
flb_http_server_client_activity_event_handler() ← FLB_ENGINE_EV_CUSTOM (non-coroutine)
→ flb_http_server_session_write()
→ flb_io_net_write()
→ flb_tls_net_write_async(co, ...) ← co == NULL (not in a coroutine context)
```

### The stall

Under mTLS, OpenSSL returns `SSL_ERROR_WANT_READ` during the write (post-handshake
authentication). `flb_tls_net_write_async` handles this as:

```c
if (ret == FLB_TLS_WANT_READ) {
io_tls_event_switch(session, MK_EVENT_READ); // re-registers fd as FLB_ENGINE_EV_THREAD
flb_coro_yield(co, FLB_FALSE); // co == NULL → undefined behaviour
goto retry_write;
}
```

Two problems occur simultaneously:

1. `io_tls_event_switch` re-registers the fd as `FLB_ENGINE_EV_THREAD`, **detaching it from
the HTTP server's `FLB_ENGINE_EV_CUSTOM` handler**. The HTTP server loses ownership of
the fd.

2. `flb_coro_yield(NULL, ...)` is called with a NULL coroutine — there is no coroutine to
resume when the read event fires.

When the socket becomes readable, the engine dispatches to the coroutine scheduler which
attempts to resume a NULL coroutine — a no-op. The response data remains stuck in the
`outgoing_data` buffer forever and the fd is orphaned.

### Why the span is stored despite the missing ACK

The span **is** stored correctly by fluent-bit (the `[INPUT]` side works). Only the gRPC ACK
(`ExportTraceServiceResponse`) is never sent.

When the gRPC client's deadline fires and sends `RST_STREAM`, the resulting fd error triggers
session teardown which flushes kernel-level TCP buffers — but by then it is too late for a
proper gRPC response.

### nghttp2 send callback is not a socket write

For additional context: `nghttp2`'s send callback (`http2_send_callback`) does **not** write
to the socket — it only appends serialized HTTP/2 frames to an in-memory `outgoing_data` SDS
buffer. The actual socket write happens later via `flb_http_server_session_write()`, which is
the point where the TLS stall occurs.

## Suggested fix directions

1. **Null-check on `co` in `flb_tls_net_write_async`** — fall back to synchronous
`flb_tls_net_write` when not in a coroutine context, avoiding the yield on a NULL coroutine.

2. **Run the HTTP server session write handler inside a coroutine** — so `flb_coro_get()`
returns a valid context when `flb_io_net_write` is called.

3. **Separate response flush from the read event** — after building the response, register the
fd for `MK_EVENT_WRITE` and flush `outgoing_data` from a dedicated write event, avoiding
the mixed read/write event ownership problem entirely.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.