Too many open files on Amazon Linux 2
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
### Describe the bug
We are facing issue of too many open files on a newly built Amazon Linux 2 AMI.
LSOF command returns a count of ~5k when there are no containers running and around 40-50k when containers spin up.
With td-agent in running state, it rises to 400-500k open files.
A lower td agent version works well on Amazon Linux 1 though, with 2-5k open files
### To Reproduce
Create a Amazon Linux 2 instance and download latest version of td-agent - 4.4.2
Keep some conf files in /etc/td-agent/ folder and run the following
lsof | wc -l
lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15
### Expected behavior
Too many open files message would be shown
lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15
67562 event_loo 15244
52503 flush_thr 15244
52503 enqueue_t 15244
984 dockerd 4312
552 auditbeat 4191
492 agent 4876
407 ruby 15244
324 container 4763
319 container 3915
235 socket_ma 15239
235 signal_th 15239
235 fluentd 15239
234 gssproxy 3417
209 ssm-agent 4568
209 amazon-ss 4132
lsof |wc -l
179205
### Your Environment
```markdown
- Fluentd version: 1.15.3
- TD Agent version: 4.4.2
- Operating system: Amazon Linux 2
- Kernel version: 4.14.301-224.520.amzn2.x86_64
```
### Your Configuration
```apache
@type tail
format json
pos_file /tmp/hcs.log
path /var/log/app/hcs/hcs_api.log
tag hcs.log
@type kafka_buffered
output_include_time true
output_include_tag true
brokers
flush_interval 3s
default_topic kafka-HelpCenterService
output_data_type json
buffer_type file
buffer_path /var/log/td-agent/buffer/hcs
buffer_chunk_limit 100m
buffer_queue_limit 256
```
### Your Error Log
```shell
Too many open files message would be shown
lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15
67562 event_loo 15244
52503 flush_thr 15244
52503 enqueue_t 15244
984 dockerd 4312
552 auditbeat 4191
492 agent 4876
407 ruby 15244
324 container 4763
319 container 3915
235 socket_ma 15239
235 signal_th 15239
235 fluentd 15239
234 gssproxy 3417
209 ssm-agent 4568
209 amazon-ss 4132
lsof |wc -l
179205
```
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.