Graylog2 / Graylog2/graylog2-server

Add global event processor delay based on maximum index refresh interval

Open
#18,913 0 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
Individual jobs should be able to consider long refresh intervals when calculating the next run time of a job trigger.

## Current Behavior
For overly long index refresh intervals, the event processor jobs must be made aware of the additional delay they must consider.
They already take into account latency induced by journal buffering but have no visibility into index refresh intervals.
Thus, it can happen that the next run of a job does not coincide with an index refresh, leading to missed events in the gap.
The default job trigger poll interval is 1 second, and the index refresh interval is 1 second, making the potential problem minimal in the default case. However, using a very large index refresh interval can become an issue.

## Possible Solution
When we rotate an index, we should retrieve the applicable index refresh interval from its settings and store this information.
Event processors should consider the refresh interval when calculating their next run time.

As long as the delay is longer than the refresh interval, we should always see the data for the time range of a job. The best value is to be determined.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.