redpanda-data / redpanda-data/connect
upgrade "files" input with watchdog
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
It would be nice to have an opportunity to use benthos instead of filebeat or rsyslog for simple shipping logs so it could expand its influence and conquer more use-cases. But currently benthos'es "files" input reads path just once, hence in order to ship new logs we have to restart the instance.
I also wonder if it has metadata in it to understand where benthos stopped its reading if we had it restarted.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the files input implementation and its tests. Read how the input discovers paths and tracks reading progress, then determine how watchdog-based updates and restart-safe metadata should behave. Done means new log data is shipped without restarting and reading resumes correctly after a restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100