Idea: add a mode where Prometheus will do head compaction/WAL truncation before starting scraping

Proposal

As @mattburgess said at https://github.com/prometheus/prometheus/issues/6934#issuecomment-945739611

would it be possible to run a compaction/WAL cleanup as soon as the replay has been completed? That would help us avoid the crash-loop we find ourselves in, whereby a pod restart causes WAL replay to happen, but then they never get cleaned up, so when the OOM-killer comes along the same WALs get replayed on the next restart and on and on we go in circles until we resign ourselves to removing the WALs and thereby losing data.

I found myself wanting this today.

I will add, I think it is important that we not start scraping until after this head compaction has finished, to keep memory usage down and avoid making the WAL any bigger, because if Prometheus OOMs again it will restart with a worse problem.

It could be a CLI flag to Prometheus, like --force-head-compaction-at-start.

Start at Prometheus startup after WAL replay completes, and trace when scraping begins. Determine how the proposed --force-head-compaction-at-start mode should trigger head compaction and WAL cleanup before scraping, while preserving the stated behavior after a restart. Done means the cleanup finishes before scraping and the crash-loop scenario is addressed without requiring manual WAL deletion.