fluent / fluent/fluent-bit

Fluent Bit OpenTelemetry output plugin unclear doc and inconsistent behavior with gRPC = on.

Open
#10,821 12 comments 0 reactions 0 assignees View on GitHub
troubleshooting
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

**Describe the bug**
The page https://docs.fluentbit.io/manual/data-pipeline/outputs/opentelemetry states "Only HTTP endpoints are supported.". Not 100% sure if gRPC falls under "HTTP endpoints" support or not. Could you please confirm? If not , then you may ignore the request below.

If gGPC is supported, then I'm experiencing an inconsistent behavior with the setting gRPC on. I constantly lose 2 batches of messages with high (lets say > 1000 messages / sec) rates. With lower rates does not happen.

**To Reproduce**
Log are clean. No warnings / errors. Silent drops.

- Steps to reproduce the problem:
Create the files as below -> Change tmp/ folder permissions so that OTEL container could write to it -> docker compose up --build -> this file will be created tmp/otel-logs.json -> convert the file _jq -c '.resourceLogs[].scopeLogs[].logRecords[]' otel-logs.json > otel-logs-ndjson.json_ -> count the lines _cat otel-logs-ndjson.json | wc -l_

1. docker-compose.yaml
services:
otel-collector:
image: otel/opentelemetry-collector:latest
container_name: otel-collector
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
- ./tmp/:/tmp/

fluent-bit:
image: fluent/fluent-bit:4.0.8-debug
container_name: fluent-bit
depends_on:
- otel-collector
volumes:
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
- ./counter.lua:/fluent-bit/etc/counter.lua:ro

2. fluent-bit.conf
[SERVICE]
Flush 0.5
Log_Level debug
storage.max_chunks_up 4096
storage.backlog.mem_limit 500M

[INPUT]
Name dummy
Dummy {"message":"hello"}
Rate 10000 # X messages per second
Samples 100000 # stop after Y messages
Threaded true

[FILTER]
Name lua
Match *
Script /fluent-bit/etc/counter.lua
Call cb_print

[OUTPUT]
Name opentelemetry
Match *
Host otel-collector
Port 4317
#Port 4318
Tls Off
Tls.verify Off
workers 4
Grpc On
Retry_Limit -1
batch_size 50

3. otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318

exporters:
debug: {}
file:
path: /tmp/otel-logs.json

service:
pipelines:
logs:
receivers: [otlp]
exporters: [debug, file]

4. counter.lua
count = 0

function cb_print(tag, ts, record)
count = count + 1
record["message"] = "hello #" .. count
return 2, ts, record
end

**Expected behavior**
I expect to see the same # of lines in tmp/otel-logs-ndjson.json as the number of messages I generated with the dummy plugin. Real behavior is that I receive 2 batches less. Batches can be in random locations in the generated stream, but always consecutive.

**Screenshots**
No screenshots.

**Your Environment**

2 images:
- sender -> fluent/fluent-bit:4.0.8-debug (pipeline: dummy -> filter to add counter values -> output (otlp)
- collector -> otel/opentelemetry-collector:latest

**Additional context**
gRPC protocol is supposed to have "advantages" over HTTP, however so far is not reliable. But http works well.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.