WAL corruption not detected on startup or erroneous WAL corruption report during Head compaction

What did you do?

Multiple WAL corruption happened due to disk running out of space. Most disk space was given.

What did you expect to see?

WAL replay path already has a WAL repair mechanism for such corruptions. I'd have expected WAL repair to identify the corruption.

What did you see instead? Under which circumstances?

WAL repair did not kick in when the WAL was corrupted for the first time. For example the corruption was for file 525, the WAL replay did not fix the corruption. But when WAL checkpointing ran after a Head compaction, the checkpoining kept erroring out because of WAL corruption in file 525. This eventually kept filling the disk (because of untruncated WAL), and it ran out of space again.

Interesting part here is, when it corrupted WAL again, for example in file 946, the WAL replay found the corruption in that file and did the repair mechanism. But the corruption in file 525, which wen't undiscovered in WAL replay, kept failing the checkpointing, causing more corruptions in the future.

This is an unrecoverable state where WAL keeps growing and filling up disk space.

System information

Linux, amd64

Prometheus version

v3.8.0

Prometheus configuration file

Alertmanager version

Alertmanager configuration file

Logs

WAL truncation in Compact: create checkpoint: read segments: corruption in segment /prometheus/wal/000005xx at 12345678: unexpected full record

Start by tracing WAL replay and its repair mechanism, then compare it with checkpointing after Head compaction using the reported corruption in segment 525 and the log message for an unexpected full record. Done means the initial corruption is detected and repaired, and checkpointing no longer leaves WAL growing until the disk fills.