in_tail plugin randomly fails with "too many open files" and errno=24 - unless switched from inotify to stat tail
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
**Describe the bug**
in_tail plugin randomly fails with "too many open files" and errno=24 - unless switched from inotify to stat tail
**To Reproduce**
- Example log message if applicable:
```
[TIMESTAMP] [ info] [storage] initializing...
[TIMESTAMP] [ info] [storage] in-memory
[TIMESTAMP] [ info] [storage] normal synchronization mode, checksum disabled
[TIMESTAMP] [ info] [engine] started (pid=14)
[TIMESTAMP] [error] [plugins/in_tail/tail_fs.c:168 errno=24] Too many open files
[TIMESTAMP] [error] Failed initialize input tail.2
[TIMESTAMP] [error] [plugins/in_tail/tail_fs.c:168 errno=24] Too many open files
[TIMESTAMP] [error] Failed initialize input tail.3
[TIMESTAMP] [error] [plugins/in_tail/tail_fs.c:168 errno=24] Too many open files
[TIMESTAMP] [error] Failed initialize input tail.4
[TIMESTAMP] [error] [plugins/in_tail/tail_fs.c:168 errno=24] Too many open files
[TIMESTAMP] [error] Failed initialize input tail.5
[TIMESTAMP] [error] [plugins/in_tail/tail_fs.c:168 errno=24] Too many open files
[TIMESTAMP] [error] Failed initialize input tail.6
[TIMESTAMP] [error] [plugins/in_tail/tail_fs.c:168 errno=24] Too many open files
[TIMESTAMP] [error] Failed initialize input tail.7
[TIMESTAMP] [error] [plugins/in_tail/tail_fs.c:168 errno=24] Too many open files
[TIMESTAMP] [error] Failed initialize input tail.8
[TIMESTAMP] [error] [plugins/in_tail/tail_fs.c:168 errno=24] Too many open files
[TIMESTAMP] [error] Failed initialize input tail.9
[TIMESTAMP] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
```
- Steps to reproduce the problem:
Happens randomly, only some pods in k8s cluster based on the Fluent-Bit docker image are affected. Ulimit on fluent-bit container:
```
/ # ulimit -S -n
1048576
/ # ulimit -H -n
1048576
```
and system max:
```
/ # cat /proc/sys/fs/file-max
13181250
```
seem fine.
But looking at https://github.com/fluent/fluent-bit/blob/v1.3.2/plugins/in_tail/tail_fs.c then at https://github.com/fluent/fluent-bit/blob/master/plugins/in_tail/tail_fs_inotify.c#L171 and https://linux.die.net/man/2/inotify_init1 - as errno=24 in that context meaning EMFILE - suggests rather inotify-related issue. Forcing cmake to use _-DFLB_INOTIFY=Off_ flag and disabling inotify and using stat tail instead is a working workaround for now. It would beneficial to also have "inotify-disabled flavour" of Docker images of Fluent-Bit available, too.
**Expected behavior**
No errors when using in_tail plugin.
**Screenshots**
n/a
**Your Environment**
* Version used: 1.0.4-1.3.3
* Configuration: Docker image, both normal and -debug version. Appears both when tailing exact file path and also when using regexp-based path (files inside a certain directory).
**Additional context**
Logs are not tailed, hence no log even processed by Fluent-Bit if that happens at given time.
Might be connected with #1712 and/or #1358
Contributor guide
Assessment
This issue has not been assessed yet.