In tail plugin, position data in pos_file often was deleted? disappear?
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
### Describe the bug
The postition information of each file in pos_file disappears.
And only empty pos_file remain.
After that, several phenomena occur.
Such as "Unparsable line in pos_file: 000000000796950c" in fluentd log,
or position information is added for files that have already been tailed, and it starts to tail again from the beginning of file.
### To Reproduce
fluentd-conf.yaml
/fluentd/source-data/ is NFS mounted server directory
```
@type tail
path "/fluentd/source-data/#{hostname}__*.log"
pos_file "/fluentd/source-data/pos_files/#{hostname}.pos"
refresh_interval 5s
follow_inodes true
skip_refresh_on_startup true
read_from_head true
read_lines_limit 10000
tag tag
pos_file_compaction_interval 1s
@type json
```
### Expected behavior
pos_file only compacts(or delete) position data when tracked file deleted or position data of duplicated file appended.
### Your Environment
```markdown
- Fluentd version: fluentd:v1.15.3-debian-1.1
- Operating system: PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
- Kernel version: Linux 4.15.0-207-generic
```
### Your Configuration
fluentd-conf.yaml
/fluentd/source-data/ is NFS mounted server directory
```apache
@type tail
path "/fluentd/source-data/#{hostname}__*.log"
pos_file "/fluentd/source-data/pos_files/#{hostname}.pos"
refresh_interval 5s
follow_inodes true
skip_refresh_on_startup true
read_from_head true
read_lines_limit 10000
tag tag
pos_file_compaction_interval 1s
@type json
```
### Your Error Log
```shell
2023-03-18 15:11:06 +0900 [warn]: #0 Unparsable line in pos_file: 000000000761339b
2023-03-18 15:11:08 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:08 +0900 [warn]: #0 Unparsable line in pos_file: 0000000006ed6a27
2023-03-18 15:11:09 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:09 +0900 [warn]: #0 Unparsable line in pos_file: 000000000726ca85
2023-03-18 15:11:09 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:11 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:11 +0900 [warn]: #0 Unparsable line in pos_file: 000000000796950c
2023-03-18 15:11:11 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:12 +0900 [info]: #0 Clean up the pos file
```
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.