fluent / fluent/fluent-operator

bug: add flush_interval_sec

Open
#1,459 2 comments 3 reactions 0 assignees View on GitHub
enhancement feature-request
Dominant language
Go
Stars
682
Forks
328
Avg merge
2d 23h
Merged PRs (30d)
13

Description

### Describe the issue

We are using

1. fluentbit image: kubesphere/fluentbit-3.1.8

2. FLuent operator version: https://github.com/fluent/fluent-operator/releases/tag/v3.2.0

We have recently implemented logToMetrics plugin using fluentbit operator CRD.
we are seeing error in our Prometheus that duplicate metrics with diffrent value and same time stamp is coming to Prometheus.
```
ts=2025-01-27T12:51:31.436Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=serviceMonitor/cloud/fluent-bit-product-01/0 target=http://10.149.29.143:2021/api/v2/metrics/prometheus msg="Error on ingesting samples with different value but same timestamp" num_dropped=16
ts=2025-01-27T12:51:32.029Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=serviceMonitor/cloud/fluent-bit-product-01/0 target=http://10.149.39.145:2021/api/v2/metrics/prometheus msg="Error on ingesting samples with different value but same timestamp" num_dropped=8
ts=2025-01-27T12:51:42.463Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=serviceMonitor/cloud/fluent-bit-product-01/0 target=http://10.149.7.132:2021/api/v2/metrics/prometheus msg="Error on ingesting samples with different value but same timestamp" num_dropped=30
ts=2025-01-27T12:51:43.822Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=serviceMonitor/cloud/fluent-bit-product-01/0 target=http://10.149.1.52:2021/api/v2/metrics/prometheus msg="Error on ingesting samples with different value but same timestamp" num_dropped=3
```
Below is our Endpoint in Service monitor:
```
endpoints:
- port: metrics
path: /api/v2/metrics/prometheus
interval: 30s
{{- end }}
```
```
Below is our log to metric plugin:
- logToMetrics:
addLabel:
- timestamp os.time()
kubernetesMode: true
metricDescription: Count of logs processed by fluent-bit - Gauge
metricMode: counter
metricName: product_log_to_metrics
tag: product_log_to_metrics
```

This issues is already opened in fluentbit end, The workaround suggested to use flush_interval_sec is not available in fluent operator CRD latest version:
https://github.com/fluent/fluent-bit/issues/9413

### To Reproduce

Same as: https://github.com/fluent/fluent-bit/issues/9413

Steps to reproduce the problem:
Deploy fluent-bit with the below tail input config as a daemonset into a k8s cluster using version 3.1.4 to see container logs and metrics to validate success.
Update fluent-bit image to 3.1.5 (or newer, <= 3.1.8) and verify /metrics endpoint on port 2021

### Expected behavior

No duplicate metrics on the additional endpoint /metrics for log_to_metrics feature usually on port 2021, no warnings in Prometheus logs, no PrometheusDuplicateTimestamps errors.

### Your Environment

```markdown
- Fluent Operator version: 3.2.0
- Environment name and version (e.g. Kubernetes? What version?): EKS Kubernetes 1.27
- Filter and Plugins: kubernetes, log_to_metrics
```

### How did you install fluent operator?

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Fluent Operator CRD schema for the logToMetrics configuration and trace how it becomes Fluent Bit configuration. Compare the requested flush_interval_sec workaround with upstream Fluent Bit issue 9413 and the reproduced metrics endpoint behavior. Done means the CRD accepts and applies the setting, with no duplicate timestamps or Prometheus warnings in the stated reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
devops, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.