fluent / fluent/fluentd

`in_tail`: `group` setting judges the limit by the average number, not the total number

Open
#4,184 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Describe the bug

https://github.com/fluent/fluentd/pull/3535#discussion_r1200099716

The `limit` of `group` setting should control the total number of lines collected for a group (https://docs.fluentd.org/input/tail#less-than-group-greater-than-section).

However, it judges the limit by the average number, not the total number.

### To Reproduce

Run Fluentd by the following command with the config below.
(`limit: 10` and `rate_limit: 1m`)

```console
$ rm -f /test/fluentd/pos/pos; echo A-1 > /test/fluentd/input/A.log; echo B-1 > /test/fluentd/input/B.log; \
bundle exec fluentd -c /test/fluentd/config/group/fluent.conf
```

After running Fluentd, run the following command on another console to add logs to `/test/fluentd/input/A.log`.

```console
$ for i in `seq 2 9`; do sleep 1; echo A-$i >> /test/fluentd/input/A.log; done;
```

After 9 seconds, the contents of these log files to be collected are as follows.

* `/test/fluentd/input/A.log`

```
A-1
A-2
A-3
A-4
A-5
A-6
A-7
A-8
A-9
```

* `/test/fluentd/input/B.log`

```
B-1
```

However, the output of Fluentd is as `Your Error Log` below.
This shows that the data is limited by the average number, not the total number.

After collecting 5 lines from the `/test/fluentd/input/A.log`, `in_tail` stops collecting.

```
2023-05-23 11:41:08.271353203 +0900 test: {"message":"A-1"}
2023-05-23 11:41:08.271511446 +0900 test: {"message":"B-1"}
# Run the command to add logs
2023-05-23 11:41:12.717511479 +0900 test: {"message":"A-2"}
2023-05-23 11:41:13.721082681 +0900 test: {"message":"A-3"}
2023-05-23 11:41:14.724453738 +0900 test: {"message":"A-4"}
2023-05-23 11:41:15.727744777 +0900 test: {"message":"A-5"}
```

After 1 minutes, `in_tail` restarts collecting.

```
2023-05-23 11:42:08.272066658 +0900 test: {"message":"A-6"}
2023-05-23 11:42:08.272076384 +0900 test: {"message":"A-7"}
2023-05-23 11:42:08.272079574 +0900 test: {"message":"A-8"}
2023-05-23 11:42:08.272082425 +0900 test: {"message":"A-9"}
```

### Expected behavior

The `limit` of `group` setting should control the total number of lines collected for a group (https://docs.fluentd.org/input/tail#less-than-group-greater-than-section).

So all 10 lines in `To Reproduce` must be collected in one period.

### Your Environment

```markdown
- Fluentd version: 1.16.1
- Operating system: Ubuntu 20.04.6 LTS
- Kernel version: 5.15.0-71-generic
```

### Your Configuration

```apache

@type tail
tag test
path /test/fluentd/input/*.log
pos_file /test/fluentd/pos/pos
read_from_head true
refresh_interval 5s

rate_period 1m
pattern /^(?.*)$/

match {"log": "/./"}
limit 10



@type none

@type stdout

```

### Your Error Log

```shell
2023-05-23 11:41:08 +0900 [info]: #0 starting fluentd worker pid=1334352 ppid=1334332 worker=0
2023-05-23 11:41:08 +0900 [info]: #0 following tail of /test/fluentd/input/A.log
2023-05-23 11:41:08.271353203 +0900 test: {"message":"A-1"}
2023-05-23 11:41:08 +0900 [info]: #0 following tail of /test/fluentd/input/B.log
2023-05-23 11:41:08.271511446 +0900 test: {"message":"B-1"}
2023-05-23 11:41:08 +0900 [info]: #0 fluentd worker is now running worker=0
2023-05-23 11:41:12.717511479 +0900 test: {"message":"A-2"}
2023-05-23 11:41:13.721082681 +0900 test: {"message":"A-3"}
2023-05-23 11:41:14.724453738 +0900 test: {"message":"A-4"}
2023-05-23 11:41:15.727744777 +0900 test: {"message":"A-5"}
2023-05-23 11:42:08.272066658 +0900 test: {"message":"A-6"}
2023-05-23 11:42:08.272076384 +0900 test: {"message":"A-7"}
2023-05-23 11:42:08.272079574 +0900 test: {"message":"A-8"}
2023-05-23 11:42:08.272082425 +0900 test: {"message":"A-9"}
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.