Graylog2 / Graylog2/graylog2-server

Server log filling up with: Couldn't read any messages from offset... journal has more messages

Open
#11,795 7 comments 1 reaction 0 assignees View on GitHub
improvement triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Customer Report
We have a three node cluster for Graylog. I started to work on upgrading them from 4.1.3 to 4.2.1 via RPM/yum repo. My process is to upgrade the node that has no inputs on it and is not a master, then the one with inputs that’s not a master, then the master. After I upgraded the software on the first node and restarted, the Graylog server.log immediately got filled with messages like:

> 2021-11-23T11:09:46.536-06:00 WARN [LocalKafkaJournal] Couldn't read any messages from offset <10372709> but journal has more messages. Skipping and trying to read from offset <10372710>

> 2021-11-23T11:09:46.536-06:00 WARN [LocalKafkaJournal] Couldn't read any messages from offset <10372710> but journal has more messages. Skipping and trying to read from offset <10372711>

HS-674468895

## Details
The customer is seeing a very large number of these messages flooding their Graylog server log file. When starting up their server, the log file just keeps rotating and the messages fill up the log file. I believe it had rotated 10 times on the last startup attempt.

## Background
In Graylog 4.1, improved handling of corrupted journal segments was implemented (see https://github.com/Graylog2/graylog2-server/pull/9378). The `Couldn't read any messages... but journal has more messages` warning message is an indication of a corrupted journal segment.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.