Fluent Bit OpenTelemetry output plugin unclear doc and inconsistent behavior with gRPC = on.
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
**Describe the bug**
The page https://docs.fluentbit.io/manual/data-pipeline/outputs/opentelemetry states "Only HTTP endpoints are supported.". Not 100% sure if gRPC falls under "HTTP endpoints" support or not. Could you please confirm? If not , then you may ignore the request below.
If gGPC is supported, then I'm experiencing an inconsistent behavior with the setting gRPC on. I constantly lose 2 batches of messages with high (lets say > 1000 messages / sec) rates. With lower rates does not happen.
**To Reproduce**
Log are clean. No warnings / errors. Silent drops.
- Steps to reproduce the problem:
Create the files as below -> Change tmp/ folder permissions so that OTEL container could write to it -> docker compose up --build -> this file will be created tmp/otel-logs.json -> convert the file _jq -c '.resourceLogs[].scopeLogs[].logRecords[]' otel-logs.json > otel-logs-ndjson.json_ -> count the lines _cat otel-logs-ndjson.json | wc -l_
1. docker-compose.yaml
services:
otel-collector:
image: otel/opentelemetry-collector:latest
container_name: otel-collector
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
- ./tmp/:/tmp/
fluent-bit:
image: fluent/fluent-bit:4.0.8-debug
container_name: fluent-bit
depends_on:
- otel-collector
volumes:
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
- ./counter.lua:/fluent-bit/etc/counter.lua:ro
2. fluent-bit.conf
[SERVICE]
Flush 0.5
Log_Level debug
storage.max_chunks_up 4096
storage.backlog.mem_limit 500M
[INPUT]
Name dummy
Dummy {"message":"hello"}
Rate 10000 # X messages per second
Samples 100000 # stop after Y messages
Threaded true
[FILTER]
Name lua
Match *
Script /fluent-bit/etc/counter.lua
Call cb_print
[OUTPUT]
Name opentelemetry
Match *
Host otel-collector
Port 4317
#Port 4318
Tls Off
Tls.verify Off
workers 4
Grpc On
Retry_Limit -1
batch_size 50
3. otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
debug: {}
file:
path: /tmp/otel-logs.json
service:
pipelines:
logs:
receivers: [otlp]
exporters: [debug, file]
4. counter.lua
count = 0
function cb_print(tag, ts, record)
count = count + 1
record["message"] = "hello #" .. count
return 2, ts, record
end
**Expected behavior**
I expect to see the same # of lines in tmp/otel-logs-ndjson.json as the number of messages I generated with the dummy plugin. Real behavior is that I receive 2 batches less. Batches can be in random locations in the generated stream, but always consecutive.
**Screenshots**
No screenshots.
**Your Environment**
2 images:
- sender -> fluent/fluent-bit:4.0.8-debug (pipeline: dummy -> filter to add counter values -> output (otlp)
- collector -> otel/opentelemetry-collector:latest
**Additional context**
gRPC protocol is supposed to have "advantages" over HTTP, however so far is not reliable. But http works well.
Contributor guide
Assessment
This issue has not been assessed yet.