fluent / fluent/fluentd

Too many open files on Amazon Linux 2

Open
#4,012 2 comments 0 reactions 0 assignees View on GitHub
waiting-for-user
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Describe the bug

We are facing issue of too many open files on a newly built Amazon Linux 2 AMI.

LSOF command returns a count of ~5k when there are no containers running and around 40-50k when containers spin up.
With td-agent in running state, it rises to 400-500k open files.

A lower td agent version works well on Amazon Linux 1 though, with 2-5k open files

### To Reproduce

Create a Amazon Linux 2 instance and download latest version of td-agent - 4.4.2
Keep some conf files in /etc/td-agent/ folder and run the following

lsof | wc -l

lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15

### Expected behavior

Too many open files message would be shown

lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15

67562 event_loo 15244
52503 flush_thr 15244
52503 enqueue_t 15244
984 dockerd 4312
552 auditbeat 4191
492 agent 4876
407 ruby 15244
324 container 4763
319 container 3915
235 socket_ma 15239
235 signal_th 15239
235 fluentd 15239
234 gssproxy 3417
209 ssm-agent 4568
209 amazon-ss 4132

lsof |wc -l
179205

### Your Environment

```markdown
- Fluentd version: 1.15.3
- TD Agent version: 4.4.2
- Operating system: Amazon Linux 2
- Kernel version: 4.14.301-224.520.amzn2.x86_64
```

### Your Configuration

```apache

@type tail
format json
pos_file /tmp/hcs.log
path /var/log/app/hcs/hcs_api.log
tag hcs.log

@type kafka_buffered
output_include_time true
output_include_tag true

brokers
flush_interval 3s

default_topic kafka-HelpCenterService
output_data_type json
buffer_type file
buffer_path /var/log/td-agent/buffer/hcs
buffer_chunk_limit 100m
buffer_queue_limit 256

```

### Your Error Log

```shell
Too many open files message would be shown

lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15

67562 event_loo 15244
52503 flush_thr 15244
52503 enqueue_t 15244
984 dockerd 4312
552 auditbeat 4191
492 agent 4876
407 ruby 15244
324 container 4763
319 container 3915
235 socket_ma 15239
235 signal_th 15239
235 fluentd 15239
234 gssproxy 3417
209 ssm-agent 4568
209 amazon-ss 4132

lsof |wc -l
179205
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.