Update in_tail to use storage plugins instead of pos_file
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
`in_tail` is using a file to record the positions of files, to restart reading from there after Fluentd process restarts.
But pos_file is formatted in custom binary, and hard to read/edit. And it requires any fixed path to be configured.
Storage plugins (especially storage_local) can store any data on paths configured via `root_dir`. It's very easy to manage, and make configuration files smaller, by reducing each paths (including pos_file) from plugin configuration sections.
`in_tail` should be updated to use storage plugins for positions.
Points to be considered:
* How to convert pos_file to storage plugin data? online update? any converter scripts?
* How to manage/clean-up old data? #1126
Contributor guide
Research direction
Start by reading the in_tail implementation and the storage plugin, especially storage_local, to understand how positions are currently written to pos_file and how root_dir-backed data is managed. Investigate the migration questions, including converting existing pos_file data and cleaning up old data related to #1126. Done means positions use storage plugins with a clear upgrade and cleanup path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100