fluent / fluent/fluent-bit

Duplicated logs using rewrite_tags as processor

Open
#11,014 4 comments 3 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

Using the `rewrite_tag` filter as processor duplicates logs for each rewrite_tag processor being added:

1. rewrite_tag emits logs once (as expected)
2. rewrite_tag emits logs twice (doubled)
3. rewrite_tag emits logs four times
4. rewrite_tag emits logs 8 times

Moving the `rewrite_tag` filters into the "old" filter section resolves the issue.

**To Reproduce**

Using the official helm Chart:

```yaml
# Chart.yaml
apiVersion: v2
dependencies:
- name: fluent-bit
repository: https://fluent.github.io/helm-charts
version: 0.54.0
- name: fluent-bit
alias: splunk-sink
repository: https://fluent.github.io/helm-charts
version: 0.54.0
- name: fluent-bit
alias: http-sink
repository: https://fluent.github.io/helm-charts
version: 0.54.0
name: fluent-bit-rewrite-tag
version: 0.0.1
```

```yaml
# values.yaml
fluent-bit: &fluent-bit-conf
image:
tag: 4.1.1

args:
- --workdir=/fluent-bit/etc
- --config=/fluent-bit/etc/conf/fluent-bit.yaml

config:
extraFiles:
fluent-bit.yaml: |
service:
flush: 1.0
log_level: info

http_server: true
hot_reload: true

pipeline:
inputs:
- name: cpu
tag: my_cpu
interval_sec: 5
processors:
logs:
- name: content_modifier
action: insert
key: "color"
value: "blue"
- name: rewrite_tag
match: my_cpu
alias: rewrite_tag-http
emitter_name: rewrite_tag-http
rule: $color .+ http true
#- name: rewrite_tag
# match: my_cpu
# alias: rewrite_tag-null1
# emitter_name: rewrite_tag-null1
# rule: $color .+ null1 true
#- name: rewrite_tag
# match: my_cpu
# alias: rewrite_tag-null2
# emitter_name: rewrite_tag-null2
# rule: $color .+ null2 true
- name: rewrite_tag
match: my_cpu
alias: rewrite_tag-splunk
emitter_name: rewrite_tag-splunk
rule: $color .+ splunk true
outputs:
- name: stdout
match: '*'
- match: http
name: http
host: rel-http-sink.default.svc
port: 9432
- match: splunk
name: splunk
host: rel-splunk-sink.default.svc
port: 9880
splunk_token: abc

http-sink: &testSinks
<<: *fluent-bit-conf

config:
extraFiles:
fluent-bit.yaml: |
service:
flush: 1.0
log_level: info

http_server: true
hot_reload: true

pipeline:
inputs:
- name: splunk
listen: 0.0.0.0
port: 9880
splunk_token: abc
- name: http
listen: 0.0.0.0
port: 9432
outputs:
- name: stdout
match: '*'

replicaCount: 1
kind: Deployment

extraPorts:
- name: http-in
port: 9432
containerPort: 9432
- name: splunk-in
port: 9880
containerPort: 9880

splunk-sink:
<<: *testSinks
```

The `http-sink` pods emits each record once, the `splunk-sink` pod twice. Un-comment the "null" rewrite_tags to further increase the log volume for the `splunk-sink`.

**Expected behavior**

Each `rewrite_tag` emits each matching record exactly once and not depending on the count of `rewrite_tag` filters before.

**Your Environment**

* Version used: fluent-bit 4.1.1, fluent-bit Helm Chart 0.54.0
* Configuration: See above
* Environment name and version: Kind k8s v1.34.0
* Server type and version:
* Operating System and version:
* Filters and plugins: `rewrite_tag` as processors

**Additional context**

We use the `rewrite_tag` filter for dynamic routing to multiple outputs. We noticed the doubled amount of logs in the Splunk sink, therefore the selected example. The implement workaround is to rollback the `rewrite_tag` filter section from `processors` in inputs to the "old" `filters` section.

E.g.
```yaml
pipeline:
inputs:
- name: cpu
tag: my_cpu
interval_sec: 5
processors:
logs:
- name: content_modifier
action: insert
key: "color"
value: "blue"
filters:
- name: rewrite_tag
match: my_cpu
alias: rewrite_tag-http
emitter_name: rewrite_tag-http
rule: $color .+ http true
#- name: rewrite_tag
# match: my_cpu
# alias: rewrite_tag-null1
# emitter_name: rewrite_tag-null1
# rule: $color .+ null1 true
#- name: rewrite_tag
# match: my_cpu
# alias: rewrite_tag-null2
# emitter_name: rewrite_tag-null2
# rule: $color .+ null2 true
- name: rewrite_tag
match: my_cpu
alias: rewrite_tag-splunk
emitter_name: rewrite_tag-splunk
rule: $color .+ splunk true
outputs:
- name: stdout
match: '*'
- match: http
name: http
host: rel-http-sink.default.svc
port: 9432
- match: splunk
name: splunk
host: rel-splunk-sink.default.svc
port: 9880
splunk_token: abc
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.