fluent / fluent/fluentd

In tail plugin, position data in pos_file often was deleted? disappear?

Open
#4,103 6 comments 0 reactions 1 assignee Claimed by @daipom View on GitHub
work-in-progress
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Describe the bug

The postition information of each file in pos_file disappears.
And only empty pos_file remain.

After that, several phenomena occur.
Such as "Unparsable line in pos_file: 000000000796950c" in fluentd log,
or position information is added for files that have already been tailed, and it starts to tail again from the beginning of file.

### To Reproduce

fluentd-conf.yaml

/fluentd/source-data/ is NFS mounted server directory
```

@type tail
path "/fluentd/source-data/#{hostname}__*.log"
pos_file "/fluentd/source-data/pos_files/#{hostname}.pos"
refresh_interval 5s
follow_inodes true
skip_refresh_on_startup true
read_from_head true
read_lines_limit 10000
tag tag
pos_file_compaction_interval 1s

@type json

```

### Expected behavior

pos_file only compacts(or delete) position data when tracked file deleted or position data of duplicated file appended.

### Your Environment

```markdown
- Fluentd version: fluentd:v1.15.3-debian-1.1
- Operating system: PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
- Kernel version: Linux 4.15.0-207-generic
```

### Your Configuration

fluentd-conf.yaml

/fluentd/source-data/ is NFS mounted server directory
```apache

@type tail
path "/fluentd/source-data/#{hostname}__*.log"
pos_file "/fluentd/source-data/pos_files/#{hostname}.pos"
refresh_interval 5s
follow_inodes true
skip_refresh_on_startup true
read_from_head true
read_lines_limit 10000
tag tag
pos_file_compaction_interval 1s

@type json

```

### Your Error Log

```shell
2023-03-18 15:11:06 +0900 [warn]: #0 Unparsable line in pos_file: 000000000761339b
2023-03-18 15:11:08 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:08 +0900 [warn]: #0 Unparsable line in pos_file: 0000000006ed6a27
2023-03-18 15:11:09 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:09 +0900 [warn]: #0 Unparsable line in pos_file: 000000000726ca85
2023-03-18 15:11:09 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:11 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:11 +0900 [warn]: #0 Unparsable line in pos_file: 000000000796950c
2023-03-18 15:11:11 +0900 [info]: #0 Clean up the pos file
2023-03-18 15:11:12 +0900 [info]: #0 Clean up the pos file
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.