fluent / fluent/fluentd

in_tail: possibly collects duplicated logs in `rotate_wait`

Open
#4,243 1 comment 0 reactions 1 assignee Claimed by @daipom View on GitHub
bug
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Describe the bug

I think the major log duplication problem about `follow_inodes` is fixed in #4237.
However, there is still a possibility of duplication (https://github.com/fluent/fluentd/pull/4237#issuecomment-1633358632).

`in_tail` possibly collects duplicated logs in `rotate_wait`.
Mulitple `TailWatcher`s can exist in `rotate_wait`, so they possibly collect the same target in that interval.

We need the mechanism not to add duplicated `TailWathcer` while detaching it.

I also wrote the code comment as follows in #4237:

https://github.com/fluent/fluentd/blob/d5685ada81ac89a35a79965f1e94bbe5952a5d3a/lib/fluent/plugin/in_tail.rb#L507-L515

This is not something like that all logs in one file are entirely duplicated.
(I believe such problems were resolved in #4237)

---

Detail explanation (from https://github.com/fluent/fluentd/pull/4237#issuecomment-1633358632)

The scenario: rotate -> update_watcher -> refresh_watcher -> `rotate_wait` elapsed

* Initial state

```
path-1: inode-b, TailWatcher-1
path-2: inode-a, TailWathcer-2
```

* Rotate happens

```
path-1: inode-c, TailWatcher-1-new
path-2: inode-b, TailWathcer-2-new, TailWatcher-1-detaching
path-3: inode-a, TailWatcher-2-detaching
```

* refresh_watcher

```
path-1: inode-c, TailWatcher-1-new
path-2: inode-b, TailWathcer-2-new, TailWatcher-1-detaching
path-3: inode-a, TailWatcher-2-detaching, TailWatcher-3
```

* `rotate_wait` elapsed

```
path-1: inode-c, TailWatcher-1-new
path-2: inode-b, TailWathcer-2-new
path-3: inode-a, TailWatcher-3
```

As above, `rotate_wait` causes the situation that multiple TailWatchers exist for the same target temporarily.
This causes log duplication.
I have confirmed this on v1.16.1.

### To Reproduce

From https://github.com/fluent/fluentd/pull/4237#issuecomment-1633358632.

Use the config below.

* Start to tail the following file

```
test.log
```

* Rotate

```
test.log
test.log.1
```

* Append a log to `test.log.1`.
* When `refresh_interval ` NOT elapsed and `rotate_wait` NOT elapsed
* No duplication
* When `refresh_interval ` elapsed and `rotate_wait` NOT elapsed
* DUPLICATION
* When `refresh_interval ` elapsed and `rotate_wait` elapsed
* No duplication

### Expected behavior

Log duplication does not occur.

### Your Environment

```markdown
- Fluentd version: 1.16.1, 1.16.2
- Operating system: Ubuntu 20.04.6 LTS
- Kernel version: 5.15.0-71-generic
```

### Your Configuration

```apache

@type tail
tag test
path /path/to/test.log*
pos_file /test/fluentd/pos/pos
follow_inodes true
refresh_interval 5s
enable_stat_watcher false # To ensure that TailWathcer recognizes rotation
rotate_wait 30s

@type none

@type stdout

```

### Your Error Log

```shell
No error.
```

### Additional context

We can set `rotate_wait 0` for a workaround.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.