fluent / fluent/fluentd

Use notification mechanisms for watching new files in tail plugin instead of refresh_interval

Open
#2,985 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.

**Is your feature request related to a problem? Please describe.**

I am using TDAgent for a couple of years already inside Kubernetes, and until now it was doing a tremendous job.
Suddenly i noticed that we are missing some logs, although from a log structure point of view everything was ok, and testing it using tdagent test kit was processing the logs just fine.

Since we had to do with an ephemeral container, that logged some error logs, and then died, i dug into the problem more, to only find out that the log file my container produced was only picked up by tdagent in_tail plugin about one minute later, which was definitely too late:

Here is the timeline :

Pod deployed (10:09:53)

```kubectl -n area11 apply -f dbmigrate.yaml && echo `date`
pod/db-migrate-9lg6q-test created
Thu May 7 10:09:53 CEST 2020```

Pod's first output (10:09:55) - so about 2-3 seconds later, my pod starts logging

`"log":"Start flyway migration\n","stream":"stdout","time":"2020-05-07T10:09:55.904911554Z"}
{"log":"2020-05-07 10:09:55 INFO - Execute_DB_Operations: DB Operation selected; Logging in /flyway_output.html\n","stream":"stdout","time":"2020-05-07T10:09:55.912142202Z"}`

Eventually the in_tail picks up the file about a minute later, and starts tailing. At this point it is too late, my pod has already logged:
```2020-05-07 10:10:51 +0000 [info]: #0 following tail of /var/log/containers/db-migrate-9lg6q-test_area11_db-migrate-0a73fe226edfa80ddc51f52756c4504625f185e220451b021bd9d67f2408efed.log```

After looking for the problem, i noticed (that is my understanding), that the only way to deal with this would be to set the refresh_interval at a very low value (like couple of seconds).

This works, indeed, if i set this to 1-3 seconds, but why??? In clusters with a lot of files, i fear that setting xx active watches will have an impact on performance.

I just cannot understand if there is anything i am missing. I was supposing that specially for stateless apps as they exist in Kubernetes, a notification mechanism, such as unix's inotifywait would be the solution to follow.

**Describe the solution you'd like**

I would like to see a notification mechanism on the tail paths, which would actively notify the tail_plugin of new appeared files, therefore minimizing to 0 the change of loosing even one single log entry
**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.