fluent / fluent/fluentd

Update in_tail to use storage plugins instead of pos_file

Open
#1,395 1 comment 0 reactions 0 assignees View on GitHub
enhancement v0.14
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

`in_tail` is using a file to record the positions of files, to restart reading from there after Fluentd process restarts.
But pos_file is formatted in custom binary, and hard to read/edit. And it requires any fixed path to be configured.

Storage plugins (especially storage_local) can store any data on paths configured via `root_dir`. It's very easy to manage, and make configuration files smaller, by reducing each paths (including pos_file) from plugin configuration sections.
`in_tail` should be updated to use storage plugins for positions.

Points to be considered:
* How to convert pos_file to storage plugin data? online update? any converter scripts?
* How to manage/clean-up old data? #1126

Contributor guide

Open the contributing guide

Research direction

Start by reading the in_tail implementation and the storage plugin, especially storage_local, to understand how positions are currently written to pos_file and how root_dir-backed data is managed. Investigate the migration questions, including converting existing pos_file data and cleaning up old data related to #1126. Done means positions use storage plugins with a clear upgrade and cleanup path.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.