fluent / fluent/fluent-bit

`fluentbit_filter_drop_records_total` metric is increased when using buffered multiline

Open
#6,699 11 comments 0 reactions 1 assignee Claimed by @RicardoAAD View on GitHub
exempt-stale long-term
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

**Describe the bug**

I'm using the multiline filter to parse go stacktrace messages and that seems to be working fine on my local minikube environment, the only issue I'm having is that the `fluentbit_filter_drop_records_total` metric is increased and cannot understand what's causing this, as illustrated below:

```
root@fluent-bit-test-4vg59:/# curl -s 127.0.0.1:2020/api/v1/metrics/prometheus
...
# HELP fluentbit_filter_drop_records_total Fluentbit metrics.
# TYPE fluentbit_filter_drop_records_total counter
fluentbit_filter_drop_records_total{name="multiline.0"} 49 1673970360696
fluentbit_filter_drop_records_total{name="record_modifier.1"} 0 1673970360696
...

root@fluent-bit-test-h6cbq:/# curl -s 127.0.0.1:2020/api/v1/metrics | jq
{
"input": {
"tail.0": {
"records": 49,
"bytes": 10291,
"files_opened": 17,
"files_closed": 1,
"files_rotated": 0
},
"emitter_for_multiline.0": {
"records": 41,
"bytes": 9756
}
},
"filter": {
"multiline.0": {
"drop_records": 49,
"add_records": 0,
"emit_records": 0
},
"record_modifier.1": {
"drop_records": 0,
"add_records": 0
}
},
"output": {
"file.0": {
"proc_records": 41,
"proc_bytes": 11027,
"errors": 0,
"retries": 0,
"retries_failed": 0,
"dropped_records": 0,
"retried_records": 0
}
}
}
```

This however does not happen when disabling buffering, but logs are not fully concatenated and can still be split across multiple log records.

**To Reproduce**

Setup a buffered multiline filter and it will automatically keep increasing the drop records. Believe this is because the records are re-emitted to the head of the pipeline, but for some reason this doesn't seem to happen for when buffering is disabled. If my assumption is correct, wouldn't it make sense to increment `fluentbit_filter_emit_records_total` instead?

**Expected behaviour**

Since the records get to the end of the pipeline and are outputted, would expect the `fluentbit_filter_drop_records_total` metric to be zero, but might be missing something here since I'm new to fluent-bit and this plugin in particular.

**Your Environment**

* fluent-bit 1.9.8 installed using fluent-bit helm chart 0.21.7
* Configuration in values.yaml:
```
annotations:
fluentbit.io/exclude: "true"

podAnnotations:
fluentbit.io/exclude: "true"

image:
repository: cr.fluentbit.io/fluent/fluent-bit
tag: "1.9.8-debug"
pullPolicy: Always

config:
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On

inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker
Tag kube.*
Mem_Buf_Limit 25MB
Skip_Long_Lines On
DB /var/log/flb_kube.db
Rotate_Wait 60

filters: |
[FILTER]
name multiline
match kube.*
multiline.key_content log
multiline.parser go
[FILTER]
Name record_modifier
Match kube.*
Record hostname ${HOSTNAME}

outputs: |
[OUTPUT]
Name file
Match kube.*
Path /tmp

logLevel: info
```
* Minikube v1.28.0

**Additional context**

The main motivation for keeping `fluentbit_filter_drop_records_total` unchanged is for helping defining alerts in prometheus, using a query such as the one below where we don't have to exclude the multiline filter which could be dropping for another reason (e.g., memory pressure):
```
rate(fluentbit_filter_drop_records_total[1m]) > 0
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.