Graylog2 / Graylog2/graylog2-server

Overutilized Journal does not get cleaned up

Open
#9,725 0 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

If the on-disk Kafka journal grows beyond `message_journal_max_size` we don't stop writing,
but try to clean up old messages to make space for new ones.
This is done with the `LogRetentionCleaner`.

Since all of this happens asynchronously, there seems to be a race.
If the journal utilization grows beyond 100%, the truncation seems to skip over some messages,
which are then never read, and never cleaned.

```
2020-12-08 17:00:18,983 WARN : org.graylog2.shared.journal.KafkaJournal - Journal utilization (1406.0%) has gone over 95%.
2020-12-08 17:00:18,983 INFO : org.graylog2.shared.journal.KafkaJournal - Journal usage is 1406.00% (threshold 100%), changing load balancer status from ALIVE to THROTTLED
2020-12-08 17:00:18,987 INFO : org.graylog2.shared.journal.KafkaJournal - Read offset 1739281 before start of log at 2422060, starting to read from the beginning of the journal.
2020-12-08 17:00:18,997 INFO : org.graylog2.shared.journal.KafkaJournal - Read offset 1739281 before start of log at 4498836, starting to read from the beginning of the journal.
```

After running stopping my benchmark, the journal utilization only went down to `69%` claiming that there are no
more messages to process.

Inspecting with the `JournalShow` command:

```
Graylog message journal in directory: /home/mpf/code/graylog-project-repos/graylog2-server/data/journal
Total size in bytes: 73216172
Number of segments: 1
Log start offset: 7257530
Log end offset: 7499348
Number of messages: 241819
Committed read offset: 7499348
Segment 1
Base offset: 7257530
Size in bytes: 73216172
Created at: 2020-12-08T16:25:40.531Z
Last modified: 2020-12-08T16:17:01.000Z
```
```
$ cat data/journal/graylog2-committed-read-offset
7499348
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.