Graylog2 / Graylog2/graylog2-server
Graylog not gracefully handling a disk full situation in Elasticsearch
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
When Elasticsearch encounters a disk full situation and an index fails shard allocation, Graylog should be aware that the index technically exists to allow its deletion. The primary index failed allocation but the replica was created but not assigned. I should be able to see and delete this from Graylog.
## Current Behavior
Graylog is attempting to access it, but doesn't even acknowledge that the index exists in the set.
## Possible Solution
As a workaround I had to destroy the index manually within Elasticsearch and recalculate index ranges, waiting some time for it to complete.
## Context
```
bro_9 0 p UNASSIGNED ALLOCATION_FAILED
bro_9 0 r UNASSIGNED INDEX_CREATED
```
```
Error Message:
Unable to perform search query Index not found for query: bro_9. Try recalculating your index ranges.
Details:
Index not found for query: bro_9. Try recalculating your index ranges.
Search status code:
500
Search response:
cannot GET https://itdgraylog.nd.gov/api/search/universal/relative?query=%2A&range=300&filter=streams%3A5b353c01b8d3e6548339cdf2&limit=150&sort=timestamp%3Adesc (500)
```

## Your Environment
* Graylog Version: 2.4.5
* Elasticsearch Version: 5.6.10-1
* MongoDB Version: 3.4.15-1.el7
* Operating System: RHEL 7.5 (Maipo)
* Browser version: Chrome 67.0.3396.99 on Windows 10
Contributor guide
Assessment
This issue has not been assessed yet.