fluent / fluent/fluent-bit

[log_to_metrics] Randomly stops outputting with `given value field is empty or not existent`

Open
#10,581 8 comments 4 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

**Describe the bug**
We have a recurring problem where the output metrics from log_to_metrics just stop with the same block of errors
```
[2025/02/21 12:05:12] [error] [filter:log_to_metrics:log_to_metrics.2] given value field is empty or not existent: $kubernetes['namespace_name']. Skipping labels.
[2025/02/21 12:05:12] [error] [filter:log_to_metrics:log_to_metrics.2] given value field is empty or not existent: $kubernetes['pod_name']. Skipping labels.
[2025/02/21 12:05:12] [error] [filter:log_to_metrics:log_to_metrics.2] given value field is empty or not existent: $kubernetes['container_name']. Skipping labels.
[2025/02/21 12:05:12] [error] [filter:log_to_metrics:log_to_metrics.2] given value field is empty or not existent: $kubernetes['docker_id']. Skipping labels.
[2025/02/21 12:05:12] [error] [filter:log_to_metrics:log_to_metrics.2] given value field is empty or not existent: $kubernetes['pod_id']. Skipping labels.
```
Restarting the fluentbit container stops the errors and resumes the output of metrics so there is nothing actually wrong with the logs in the first place.

**To Reproduce**
Unknown how to reproduce as it just happens for us and continues that way until restart but below is our config

```
data:
custom_parsers.conf: |
[PARSER]
Name cri-custom
Format Regex
Regex ^(?

[INPUT]
Name tail
Path /var/log/containers/*.log
Tag kube.*
Parser cri-custom
Mem_Buf_Limit 10MB
Skip_Long_Lines On
Read_from_Head True
DB /var/log/fluent-bit-tail-db

[INPUT]
Name systemd
Tag host.*
Read_From_Tail Off
Mem_Buf_Limit 10MB
Strip_Underscores On
DB /var/log/fluent-bit-systemd-db

[FILTER]
Name kubernetes
Match kube.var.log.containers.*
Merge_Log On
Merge_Log_Key applog
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
Annotations Off

[FILTER]
Name modify
Match host.*
Add NODE ${NODE_NAME}

[FILTER]
Name grep
Match kube.var.log.containers.*
Exclude log /error registering chunk with tag/

[FILTER]
Name log_to_metrics
Match kube.var.log.containers.*
tag log_count
add_label app $kubernetes['labels']['app.kubernetes.io/name']
add_label costcentre $kubernetes['labels']['platform.turtles.uk/costcentre']
metric_mode counter
metric_name log_count
metric_description This metric shows the count of log events
kubernetes_mode on
# Required due to https://github.com/fluent/fluent-bit/issues/9413
Flush_Interval_Sec 10

[FILTER]
Name modify
Match kube.var.log.containers.*platform-tools_event-exporter*
Rename applog events
[FILTER]
Name modify
Match kube.var.log.containers.*istio-proxy*
Rename applog istio
[FILTER]
Name rewrite_tag
Emitter_Mem_Buf_Limit 20MB
Match kube.var.log.containers.*
Rule $kubernetes['namespace_name'] .* kube.$kubernetes['namespace_name'] false

[FILTER]
Name modify
Match *
Add bosun_env prd
Add bosun_cluster prd-ie-01
Add bosun_cluster_type cluster

# This will be a lot nicer when we support YAML config
[FILTER]
Name lua
Match *
call strip_colours
code function strip_colours(tag, timestamp, record) if type(record["log"]) == "string" then; record["log"] = record["log"]:gsub('\27[[0-9;]*[mGKHF]', ''); return 2, 0, record; end; return 0, 0, 0; end

[OUTPUT]
Name kafka
Match *
Brokers :32000
Topics logs-os
timestamp_format iso8601
rdkafka.security.protocol sasl_ssl
rdkafka.enable.ssl.certificate.verification true
rdkafka.sasl.mechanism SCRAM-SHA-512
rdkafka.sasl.username logs-user-writer
rdkafka.sasl.password ${KAFKA_USER_WRITER}

[OUTPUT]
name prometheus_exporter
match *
host 0.0.0.0
port 2021
```

**Expected behavior**
It not break

**Your Environment**
* Version used: 3.2.10
* Configuration: see above
* Environment name and version (e.g. Kubernetes? What version?): eks 1.32
* Server type and version: bottlerocket 1.39.1
* Operating System and version: bottlerocket 1.39.1
* Filters and plugins: See above

Contributor guide

Open the contributing guide

Research direction

Start with the log_to_metrics filter and the supplied Kubernetes filter configuration, then trace how the label fields are read after output stops. Use the reported error sequence and restart behavior to narrow down the state change. Done means metrics continue emitting with the configured labels without requiring a Fluent Bit restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, kubernetes, prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.