Extend cursor/checkpoint option to additional input plugins.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
Similar functionality to this request currently exists in the Jdbc input plugin with the two configuration options:
```
record_last_run
last_run_metadata_path
```
Similar options exist in others such as various database input plugins and the imap input plugin.
```
sincedb_path
sequence_path
```
Indirectly, the Event Hubs input also does this (outside of the config) as it writes a checkpoint to a storage account.
Having this option, likely using the time of the last successful run, would be a great enhancement to some of the other pull-based plugins in terms of avoiding/minimizing deduplication and more resiliency should an input fail during any number of attempts.
A few examples:
rss - Each run generally needs to be deduplicated and this is an issue brought up on [this Elastic Blog post](https://www.elastic.co/blog/how-to-leverage-rss-feeds-to-inform-the-possibilities-with-elastic-stack). At minimum, being able to drop something published/updated after the cursor timestamp (or other field that can be iterated on) would cut down the processing overhead.
http_poller - similar to rss this can filter off the timestamp. Referencing it (an environmental variable?) in the http query itself will work with any api calls. Though I could see how that could be a larger change. Looking at the m365 defender integration for elastic agent, it is also using a timestamp cursor after pulling events.
These would likely hold true for most inputs where Logstash itself is pulling events.
Contributor guide
Assessment
This issue has not been assessed yet.