Graylog2 / Graylog2/graylog2-server
Disk journal stopped working after disk filled
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
Disk journal should resume processing queued messages
## Current Behavior
Processing was paused and messages kept queuing in disk journal.
## Possible Solution
unknown
## Steps to Reproduce (for bugs)
1. Let the disk fill to 100%
2. Clear some space manually via SSH terminal
3. Restart server
4. Check Processing Status and Disk Journal under System -> Node -> Details and check the server logs.
## Context
After the disk filled to 100% (misconfig from me) and the issue was fixed, server was restarted but disk journal messages processing did not resume.
I can add the web and API interface stopped responding when disk was full because mongoDB could not launch.
By looking at the server logs, I get this error:
```
2016-08-04_14:20:54.49828 2016-08-04 10:20:54,494 ERROR: com.google.common.util.concurrent.ServiceManager - Service JournalReader [FAILED] has failed in the RUNNING state.
2016-08-04_14:20:54.49830 java.lang.IllegalStateException: Invalid message size: -897035486
2016-08-04_14:20:54.49830 at kafka.log.FileMessageSet.searchFor(FileMessageSet.scala:127) ~[graylog.jar:?]
2016-08-04_14:20:54.49830 at kafka.log.LogSegment.translateOffset(LogSegment.scala:105) ~[graylog.jar:?]
2016-08-04_14:20:54.49830 at kafka.log.LogSegment.read(LogSegment.scala:147) ~[graylog.jar:?]
2016-08-04_14:20:54.49831 at kafka.log.Log.read(Log.scala:443) ~[graylog.jar:?]
2016-08-04_14:20:54.49831 at org.graylog2.shared.journal.KafkaJournal.read(KafkaJournal.java:462) ~[graylog.jar:?]
2016-08-04_14:20:54.49831 at org.graylog2.shared.journal.KafkaJournal.read(KafkaJournal.java:435) ~[graylog.jar:?]
2016-08-04_14:20:54.49831 at org.graylog2.shared.journal.JournalReader.run(JournalReader.java:136) ~[graylog.jar:?]
2016-08-04_14:20:54.49831 at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60) [graylog.jar:?]
2016-08-04_14:20:54.49831 at com.google.common.util.concurrent.Callables$3.run(Callables.java:100) [graylog.jar:?]
2016-08-04_14:20:54.49832 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
```
## Your Environment
- Graylog Version: 2.0.3 (OVA image)
- Elasticsearch Version:
- MongoDB Version:
- Operating System:
- Browser version:
Contributor guide
Assessment
This issue has not been assessed yet.