elastic / elastic/logstash

Extend cursor/checkpoint option to additional input plugins.

Open
#15,872 0 comments 0 reactions 1 assignee Claimed by @flexitrev View on GitHub
enhancement status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Similar functionality to this request currently exists in the Jdbc input plugin with the two configuration options:
```
record_last_run
last_run_metadata_path

```
Similar options exist in others such as various database input plugins and the imap input plugin.

```
sincedb_path
sequence_path
```
Indirectly, the Event Hubs input also does this (outside of the config) as it writes a checkpoint to a storage account.

Having this option, likely using the time of the last successful run, would be a great enhancement to some of the other pull-based plugins in terms of avoiding/minimizing deduplication and more resiliency should an input fail during any number of attempts.

A few examples:

rss - Each run generally needs to be deduplicated and this is an issue brought up on [this Elastic Blog post](https://www.elastic.co/blog/how-to-leverage-rss-feeds-to-inform-the-possibilities-with-elastic-stack). At minimum, being able to drop something published/updated after the cursor timestamp (or other field that can be iterated on) would cut down the processing overhead.

http_poller - similar to rss this can filter off the timestamp. Referencing it (an environmental variable?) in the http query itself will work with any api calls. Though I could see how that could be a larger change. Looking at the m365 defender integration for elastic agent, it is also using a timestamp cursor after pulling events.

These would likely hold true for most inputs where Logstash itself is pulling events.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.