fluent / fluent/fluentd

Fluentd forwarder status page is displayed with a huge delay when aggregator node is responding slowly

Open
#2,137 3 comments 0 reactions 1 assignee Claimed by @repeatedly View on GitHub
bug v1
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

OS: centos 7
Fluentd version: td-agent-3.2.0-0.el7.x86_64

When aggregator node is failing or responding very slowly while under heavy load, it might take up to 1-2 minutes to get a status page /api/plugins.json on a forwarder node.

Steps to reproduce

Forwarder config

```

@type monitor_agent
bind 127.0.0.1
port 24220

@type forward
bind 127.0.0.1
port 24224

@type forward

heartbeat_type tcp
send_timeout 60s
recover_wait 10s
heartbeat_interval 1s
# increased this while testing
phi_threshold 160000
hard_timeout 120s


name logs1
host 172.31.3.5
port 8889
weight 60

flush_interval 10s

buffer_type file
buffer_path /var/log/fluentd/buffer/forward
buffer_chunk_limit 4m
buffer_queue_limit 4096
num_threads 2
expire_dns_cache 600

```

I make some service send logs to the forwarder.

Then on aggregator node I execute

`# iptables -A INPUT -m statistic --mode random --probability 0.8 --source forwarder.node.ip.address -j DROP`

On the forwarder node I execute the following curl request in a loop

`# while true; do timeout 2 curl -s http://localhost:24220/api/plugins.json > /dev/null && echo ok || echo failure; sleep 1; done
`
In some time it starts showing "failure".

When I flush iptables rules on the aggregator node with

`iptables -F`

it gets back to normal.

It happens not all the time, but in a rather big percentage of cases it happens.

td-agent 2.5 is not affected.

Also I noticed that docker services that send logs to the forwarder stop responding sometimes as well. But was not able to reproduce it yet in my test environment.

Thanks.

Regards,
Sergey

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.