out_kafka early ack semantic introduces surprising durability characteristic
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
**Describe the bug**
The `out_kafka` plugin returns `FLB_OK` (chunk delivered) as soon as messages are enqueued into librdkafka's internal in-memory queue via `rd_kafka_produce()`. The delivery report callback (`cb_kafka_msg`) only logs warnings on failure — it does not feed back into the flush return code. This means Fluent Bit removes the chunk from its filesystem buffer before the Kafka broker has actually acknowledged the write.
If librdkafka or the broker fails after enqueue but before the message reaches Kafka, data is silently lost. The engine believes the chunk was delivered, so no retry is attempted.
This is not explicitly a bug, it can be determined by digging into the implementation details of fluent bit and librdkafka. However, it was not obvious and surprised me in testing Fluent Bit as a form of durable buffer in front of Kafka in an event processing system.
**To Reproduce**
1. Configure Fluent Bit with filesystem storage on input and an out_kafka output pointed at a Kafka broker.
2. Send data through the pipeline.
3. While messages are in-flight (enqueued in librdkafka but not yet acked by the broker), kill the Kafka broker or introduce a network partition.
4. Observe that Fluent Bit logs delivery failure warnings from the callback, but the chunk has already been removed from the filesystem buffer and is never retried.
**Expected behavior**
The plugin should have an option to block the flush until broker acknowledgment is received, and return `FLB_RETRY` if delivery fails, so the engine can re-send the chunk. This would extend Fluent Bit's at-least-once delivery guarantee through to the Kafka broker, not just to librdkafka's in-memory queue.
Alternatively, this behaviour should be clearly documented if supporting true at-least-once delivery is out of scope.
**Your Environment**
* Version used: 4.2.2
* Configuration:
```yaml
service:
flush: 1
log_level: info
daemon: off
health_check: on
storage.sync: full
storage.checksum: on
storage.max_chunks_up: 128
storage.backlog.mem_limit: 200M
pipeline:
inputs:
- name: http
listen: 0.0.0.0
port: 8888
threaded: true
storage.type: filesystem
storage.pause_on_chunks_overlimit: off
outputs:
- name: kafka
match: "*"
brokers:
timestamp_key: ts_ingestion
timestamp_format: iso8601_ns
retry_limit: false
rdkafka.enable.idempotence: false
rdkafka.compression.type: lz4
rdkafka.batch.size: 16777216
rdkafka.batch.num.messages: 100000
rdkafka.linger.ms: 100
rdkafka.max.in.flight.requests.per.connection: 1000000
rdkafka.socket.send.buffer.bytes: 0
rdkafka.socket.receive.buffer.bytes: 0
rdkafka.queue.buffering.max.messages: 1000
rdkafka.queue.buffering.max.kbytes: 16384
rdkafka.request.required.acks: -1
rdkafka.message.timeout.ms: 10000
rdkafka.socket.timeout.ms: 1000
rdkafka.metadata.request.timeout.ms: 1000
rdkafka.topic.metadata.refresh.interval.ms: 5000
```
* Environment name and version: Kubernetes 1.33
* Operating System and version: using the `cr.fluentbit.io/fluent/fluent-bit:4.2.2` image
* Server type and version:
**Additional context**
This affects any pipeline where data loss is unacceptable (audit logs, financial events, compliance data). Filesystem buffering and retry logic provide a false sense of durability without this fix — the engine considers the chunk delivered when it has only been enqueued locally in librdkafka's memory.
Contributor guide
Assessment
This issue has not been assessed yet.