Graylog2 / Graylog2/graylog2-server

Racecondition on Index Cycle (manual or by retention config) with IndexBlockCheck

Open
#19,571 2 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

Errors under System/Overview are reliable
## Current Behavior

Cycling an index in an env with hundreds of index sets leads to an Indices blocked error under System/Overview for the _old_ Index. It seams that the check (https://github.com/Graylog2/graylog2-server/blob/5.2.8/graylog2-server/src/main/java/org/graylog2/periodical/IndexBlockCheck.java) checks the old index rather then the new one.
## Possible Solution

https://github.com/Graylog2/graylog2-server/blob/5.2.8/graylog2-server/src/main/java/org/graylog2/periodical/IndexBlockCheck.java#L59

Currently fetches all write indices and afterwards loop over all to check.
Better approach could be to loop over all index sets and in this process fetch the current index and check imediatly _or_ just check the write deflector. This could save up round trips to mongodb and for all index sets just the _deflector index can be checked.

Also it would be nice if this job would not run every 30 seconds as it puts unneeded pressure on the leader and can lead to flapping leader (No Leader in Cluster, short after this notification the leader reoccurs)

## Steps to Reproduce (for bugs)

1. create huge amount of index sets (maybe 500?)
2. cycle an index
3. observe sometimes false positiv alerts for blocked indices
4. in the next run of the job the alert gets cleared

## Context

## Your Environment

* Graylog Version: 5.2.x
* Java Version: 17
* OpenSearch Version: 2.12
* MongoDB Version:5
* Operating System:RHEL
* Browser version:Firefox ESR latest

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.