Graylog2 / Graylog2/graylog2-server

Graylog not gracefully handling a disk full situation in Elasticsearch

Open
#4,892 0 comments 0 reactions 0 assignees View on GitHub
elasticsearch feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
When Elasticsearch encounters a disk full situation and an index fails shard allocation, Graylog should be aware that the index technically exists to allow its deletion. The primary index failed allocation but the replica was created but not assigned. I should be able to see and delete this from Graylog.

## Current Behavior
Graylog is attempting to access it, but doesn't even acknowledge that the index exists in the set.

## Possible Solution
As a workaround I had to destroy the index manually within Elasticsearch and recalculate index ranges, waiting some time for it to complete.

## Context

```
bro_9 0 p UNASSIGNED ALLOCATION_FAILED
bro_9 0 r UNASSIGNED INDEX_CREATED
```

```
Error Message:
Unable to perform search query Index not found for query: bro_9. Try recalculating your index ranges.
Details:
Index not found for query: bro_9. Try recalculating your index ranges.
Search status code:
500
Search response:
cannot GET https://itdgraylog.nd.gov/api/search/universal/relative?query=%2A&range=300&filter=streams%3A5b353c01b8d3e6548339cdf2&limit=150&sort=timestamp%3Adesc (500)
```

![image](https://user-images.githubusercontent.com/21294728/42455565-3c728330-8358-11e8-9f38-4801c377f879.png)

## Your Environment
* Graylog Version: 2.4.5
* Elasticsearch Version: 5.6.10-1
* MongoDB Version: 3.4.15-1.el7
* Operating System: RHEL 7.5 (Maipo)
* Browser version: Chrome 67.0.3396.99 on Windows 10

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.