High CPU usage (when idle) when there are multiple out_foward block with multiple downstream nodes
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
Fluentd version 0.14.20
OS: centos7
AWS EC2 m4.large
Problem: CPU usage is very high. The fluentd only setup with 1 worker and the usage is consistently >90% on 1 CPU core even when there's no traffic at all.
I notice the CPU usage increase as the number of block increase or the number of dowstream nodes increase.
My fluentd config is as below:
```
root_dir /var/log/fluentd
@type forward
port 24226
chunk_size_limit 128M
self_hostname "receiver-1"
shared_key "xxxxxx"
cert_path ""
private_key_path ""
@type copy
@id data_1_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_2_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_3_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_4_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_5_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@id data_5_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_2-1
port 24224
host downstream_2-2
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_6_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@id data_6_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_2-1
port 24224
host downstream_2-2
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_7_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@id data_7_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_2-1
port 24224
host downstream_2-2
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_8_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_9_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@id data_9_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_2-1
port 24224
host downstream_2-2
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_10_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@id data_10_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_2-1
port 24224
host downstream_2-2
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_11_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@id data_11_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_2-1
port 24224
host downstream_2-2
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type copy
@id data_12_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_1-1
port 24224
host downstream_1-2
port 24224
host downstream_1-3
port 24224
host downstream_1-4
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@id data_12_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""
self_hostname "receiver-1"
shared_key "yyyyyy"
host downstream_2-1
port 24224
host downstream_2-2
port 24224
@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736
@type null
```
Contributor guide
Research direction
Start by reproducing the idle CPU usage with the supplied Fluentd 0.14.20 configuration, then compare CPU as the number of match blocks, stores, and downstream nodes changes. Done means the cause is identified and idle CPU no longer scales unexpectedly with those configuration elements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- backend, networking, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100