Duplicated logs using rewrite_tags as processor
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
Using the `rewrite_tag` filter as processor duplicates logs for each rewrite_tag processor being added:
1. rewrite_tag emits logs once (as expected)
2. rewrite_tag emits logs twice (doubled)
3. rewrite_tag emits logs four times
4. rewrite_tag emits logs 8 times
Moving the `rewrite_tag` filters into the "old" filter section resolves the issue.
**To Reproduce**
Using the official helm Chart:
```yaml
# Chart.yaml
apiVersion: v2
dependencies:
- name: fluent-bit
repository: https://fluent.github.io/helm-charts
version: 0.54.0
- name: fluent-bit
alias: splunk-sink
repository: https://fluent.github.io/helm-charts
version: 0.54.0
- name: fluent-bit
alias: http-sink
repository: https://fluent.github.io/helm-charts
version: 0.54.0
name: fluent-bit-rewrite-tag
version: 0.0.1
```
```yaml
# values.yaml
fluent-bit: &fluent-bit-conf
image:
tag: 4.1.1
args:
- --workdir=/fluent-bit/etc
- --config=/fluent-bit/etc/conf/fluent-bit.yaml
config:
extraFiles:
fluent-bit.yaml: |
service:
flush: 1.0
log_level: info
http_server: true
hot_reload: true
pipeline:
inputs:
- name: cpu
tag: my_cpu
interval_sec: 5
processors:
logs:
- name: content_modifier
action: insert
key: "color"
value: "blue"
- name: rewrite_tag
match: my_cpu
alias: rewrite_tag-http
emitter_name: rewrite_tag-http
rule: $color .+ http true
#- name: rewrite_tag
# match: my_cpu
# alias: rewrite_tag-null1
# emitter_name: rewrite_tag-null1
# rule: $color .+ null1 true
#- name: rewrite_tag
# match: my_cpu
# alias: rewrite_tag-null2
# emitter_name: rewrite_tag-null2
# rule: $color .+ null2 true
- name: rewrite_tag
match: my_cpu
alias: rewrite_tag-splunk
emitter_name: rewrite_tag-splunk
rule: $color .+ splunk true
outputs:
- name: stdout
match: '*'
- match: http
name: http
host: rel-http-sink.default.svc
port: 9432
- match: splunk
name: splunk
host: rel-splunk-sink.default.svc
port: 9880
splunk_token: abc
http-sink: &testSinks
<<: *fluent-bit-conf
config:
extraFiles:
fluent-bit.yaml: |
service:
flush: 1.0
log_level: info
http_server: true
hot_reload: true
pipeline:
inputs:
- name: splunk
listen: 0.0.0.0
port: 9880
splunk_token: abc
- name: http
listen: 0.0.0.0
port: 9432
outputs:
- name: stdout
match: '*'
replicaCount: 1
kind: Deployment
extraPorts:
- name: http-in
port: 9432
containerPort: 9432
- name: splunk-in
port: 9880
containerPort: 9880
splunk-sink:
<<: *testSinks
```
The `http-sink` pods emits each record once, the `splunk-sink` pod twice. Un-comment the "null" rewrite_tags to further increase the log volume for the `splunk-sink`.
**Expected behavior**
Each `rewrite_tag` emits each matching record exactly once and not depending on the count of `rewrite_tag` filters before.
**Your Environment**
* Version used: fluent-bit 4.1.1, fluent-bit Helm Chart 0.54.0
* Configuration: See above
* Environment name and version: Kind k8s v1.34.0
* Server type and version:
* Operating System and version:
* Filters and plugins: `rewrite_tag` as processors
**Additional context**
We use the `rewrite_tag` filter for dynamic routing to multiple outputs. We noticed the doubled amount of logs in the Splunk sink, therefore the selected example. The implement workaround is to rollback the `rewrite_tag` filter section from `processors` in inputs to the "old" `filters` section.
E.g.
```yaml
pipeline:
inputs:
- name: cpu
tag: my_cpu
interval_sec: 5
processors:
logs:
- name: content_modifier
action: insert
key: "color"
value: "blue"
filters:
- name: rewrite_tag
match: my_cpu
alias: rewrite_tag-http
emitter_name: rewrite_tag-http
rule: $color .+ http true
#- name: rewrite_tag
# match: my_cpu
# alias: rewrite_tag-null1
# emitter_name: rewrite_tag-null1
# rule: $color .+ null1 true
#- name: rewrite_tag
# match: my_cpu
# alias: rewrite_tag-null2
# emitter_name: rewrite_tag-null2
# rule: $color .+ null2 true
- name: rewrite_tag
match: my_cpu
alias: rewrite_tag-splunk
emitter_name: rewrite_tag-splunk
rule: $color .+ splunk true
outputs:
- name: stdout
match: '*'
- match: http
name: http
host: rel-http-sink.default.svc
port: 9432
- match: splunk
name: splunk
host: rel-splunk-sink.default.svc
port: 9880
splunk_token: abc
```
Contributor guide
Assessment
This issue has not been assessed yet.