tail: offset_key inconsistent with stored offset after restart
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
### Describe the bug
`offset_key` in the `tail` input plugin reports offsets starting from 0 for newly appended lines after Fluent Bit restart, even though the SQLite database stores the correct absolute offset. The file content is read from the correct position, but the `offset_key` value attached to records is wrong.
This causes data corruption in downstream systems that use `offset_key` for positional writes.
### To Reproduce
1. Configure `tail` input with `offset_key` and `db` enabled
2. Let Fluent Bit read a file partially (e.g. DB stores offset 9,840,308)
3. Restart Fluent Bit
4. Append new lines to the file
5. Observe `offset_key` values in output records: 0, 143, 286... instead of 9840308, 9840451, ...
### Expected behavior
`offset_key` should report absolute byte offset from the beginning of the file, consistent with the offset stored in the database.
### Your Environment
- **Fluent Bit version:** v5.0.0, v5.0.1, v4.2.0 (confirmed on all three)
- **Operating System:** Ubuntu 22.04 (Docker)
- **Configuration file:**
```yaml
pipeline:
inputs:
- name: tail
tag: test
path: /mnt/input/*.log
path_key: file_path
offset_key: file_offset
read_from_head: true
refresh_interval: 5
inotify_watcher: false
skip_empty_lines: false
skip_long_lines: false
mem_buf_limit: 30Mb
buffer_chunk_size: 32Kb
buffer_max_size: 2Mb
threaded: true
static_batch_size: 5Mb
event_batch_size: 5Mb
db: /fluent-bit/data/test.db
outputs:
- name: tcp
match: test
host: receiver
port: 5170
format: json_lines
workers: 1
retry_limit: false
net.keepalive: true
```
### Evidence (captured via tcpdump on TCP output)
**DB stores the correct absolute offset:**
```
sqlite> SELECT * FROM in_tail_files;
3|/mnt/input/test.log|9840308|33|1774994106|0
```
**Before fix (bug) -- offsets start from 0 instead of 9,840,308 after restart:**
```json
{"file_path":"/mnt/input/test.log","file_offset":0,"log":"..."}
{"file_path":"/mnt/input/test.log","file_offset":143,"log":"..."}
{"file_path":"/mnt/input/test.log","file_offset":286,"log":"..."}
```
**After fix -- offsets match the DB position:**
```json
{"file_path":"/mnt/input/test.log","file_offset":9840308,"log":"..."}
{"file_path":"/mnt/input/test.log","file_offset":9840451,"log":"..."}
{"file_path":"/mnt/input/test.log","file_offset":9840594,"log":"..."}
```
Contributor guide
Assessment
This issue has not been assessed yet.