fluent / fluent/fluentd

High CPU usage (when idle) when there are multiple out_foward block with multiple downstream nodes

Open
#1,852 0 comments 1 reaction 0 assignees View on GitHub
v1
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

Fluentd version 0.14.20
OS: centos7
AWS EC2 m4.large

Problem: CPU usage is very high. The fluentd only setup with 1 worker and the usage is consistently >90% on 1 CPU core even when there's no traffic at all.
I notice the CPU usage increase as the number of block increase or the number of dowstream nodes increase.
My fluentd config is as below:

```

root_dir /var/log/fluentd

@type forward
port 24226
chunk_size_limit 128M

self_hostname "receiver-1"
shared_key "xxxxxx"


cert_path ""
private_key_path ""

@type copy


@id data_1_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_2_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_3_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_4_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_5_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736


@id data_5_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_2-1
port 24224


host downstream_2-2
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_6_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736


@id data_6_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_2-1
port 24224


host downstream_2-2
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_7_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736


@id data_7_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_2-1
port 24224


host downstream_2-2
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_8_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_9_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736


@id data_9_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_2-1
port 24224


host downstream_2-2
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_10_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736


@id data_10_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_2-1
port 24224


host downstream_2-2
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_11_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736


@id data_11_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_2-1
port 24224


host downstream_2-2
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type copy


@id data_12_downstream_1
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_1-1
port 24224


host downstream_1-2
port 24224


host downstream_1-3
port 24224


host downstream_1-4
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736


@id data_12_downstream_2
@type forward
require_ack_response true
transport tls
tls_cert_path ""


self_hostname "receiver-1"
shared_key "yyyyyy"


host downstream_2-1
port 24224


host downstream_2-2
port 24224


@type file
flush_mode interval
flush_interval 2s
retry_forever true
retry_type periodic
retry_wait 60s
chunk_limit_size 67108864
total_limit_size 68719476736

@type null

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the idle CPU usage with the supplied Fluentd 0.14.20 configuration, then compare CPU as the number of match blocks, stores, and downstream nodes changes. Done means the cause is identified and idle CPU no longer scales unexpectedly with those configuration elements.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
backend, networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.