in_tail: Unable to detect rotation of the log file older than ignore_older
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
**Describe the bug**
If an old log file older than `ignore_older` already exists when fluent-bit starts, and then the old log file is rotated and a new log file is created, fluent-bit will skip the beginning of the new log file.
**To Reproduce**
- Steps to reproduce the problem:
1. set variable
```
# IGNORE_OLDER=60
# REFRESH_INTERVAL=10
```
2. create log file older than `ignore_older`
```
# echo 001 > /tmp/input.log
# echo 002 >> /tmp/input.log
# touch --date "$((IGNORE_OLDER + 5 )) seconds ago" /tmp/input.log
```
3. start fluent-bit
```
# systemctl start fluent-bit
```
4. wait refresh interval
```
# sleep $(( $REFRESH_INTERVAL + 1 ))
```
5. rename (rotation)
```
# mv /tmp/input.log /tmp/input.log.1
```
6. create a new file for input
```
# echo 003 > /tmp/input.log; \
echo 004 >> /tmp/input.log; \
echo 005 >> /tmp/input.log
```
7. wait refresh interval
```
# sleep $(( $REFRESH_INTERVAL + 1 ))
```
8. stop fluent-bit
```
# systemctl stop fluent-bit
```
**Expected behavior**
expected output (/tmp/output.log)
```
{"log":"003"}
{"log":"004"}
{"log":"005"}
```
**Actual results**
actual output (/tmp/output.log)
```
{"log":"005"}
```
**Your Environment**
* Version used: fluent-bit-5.0.5-1.x86_64
* Configuration:
```
# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
Flush 1
Daemon off
Log_Level debug
Parsers_file /etc/fluent-bit/parsers.conf
storage.backlog.flush_on_shutdown on
[INPUT]
Name tail
Path /tmp/input.log
Read_From_Head true
read_newly_discovered_files_from_head true
Parser not_empty_line
db /tmp/input-tail.db
Ignore_Older 60
ignore_active_older_files true
refresh_interval 10
[OUTPUT]
Name file
File /tmp/output.log
Format plain
Match *
```
```
# /etc/fluent-bit/parsers.conf
[PARSER]
Name not_empty_line
Format regex
Regex ^.+$
```
* Environment name and version (e.g. Kubernetes? What version?): systemd 239 (239-82.el8_10.16)
* Server type and version: VM
* Operating System and version: Rocky Linux release 8.10 (Green Obsidian)
* Filters and plugins:
**Additional context**
This bug was introduced in commit 7d0c4c3ecda5a7682ccda9e40982d3ff92106ae8.
Contributor guide
Assessment
This issue has not been assessed yet.