Size retention not done when block compaction fails

What did you do?

I configured Prometheus with a size retention corresponding to 90% of disk storage, 3865470566 bytes for a disk of 4Gb, and 10 days of retention.

What did you expect to see?

I expected that block cleanup should be done if used storage was above the size retention.

What did you see instead? Under which circumstances?

I see that Prometheus is first trying to do some block compaction. That block compaction is failing: msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"

And retention by size is not done because block compaction is failing

Block inventory:

promtool tsdb list /prometheus
BLOCK ULID                  MIN TIME       MAX TIME       DURATION       NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01G5D2ATXR904DJZMR3C3VQQMZ  1654992000415  1655056800000  17h59m59.585s  381796836    3165305      134524       565458774
01G5F04DS92ESVEH2FVK06JT5K  1655056800292  1655121600000  17h59m59.708s  383120673    3160584      134356       567118293
01G5GXY0EXGENHDQ03KRJYGDXC  1655121600337  1655186400000  17h59m59.663s  384474142    3189894      134808       569025496
01G5KQ6DS25PCHR2SPHV6VRGPW  1655186400294  1655251200000  17h59m59.706s  367145990    3053079      134622       547821222
01G5MSH13DF99RZY6HAY57THT1  1655282368691  1655316000000  9h20m31.309s   201420906    1616363      134816       311242533
01G5NE40RTZP359CZPS8DTKJFF  1655316000036  1655337600000  5h59m59.964s   129411114    1070896      134898       205749441
01G5N784EQ4BE2Q6GXMW5F8RDA  1655337600581  1655344800000  1h59m59.419s   43191817     357495       135459       80464387
01G5NE3VPFBHJVEYA9MDA4709C  1655344800166  1655352000000  1h59m59.834s   43180671     357191       134839       81225973
01G5NMZJYPGNQQSSD03G5N230D  1655352000581  1655359200000  1h59m59.419s   43128282     354852       134814       80248355 

So retention size is 3865470566 bytes
Used storage is 3655376896 bytes for this block compaction, less than retention size
Free storage is 505044992 bytes, less than the storage needed for a block of 18h of data

I guess that the compaction is trying to make a block of 24h of data (10% of retention time which seems to be the maximum for the block time range) but there is not enough space.

After this first failed block compaction, other compaction launched every minutes fail, disk storage keeps growing up to nearly 100% of disk used.

My conclusion is that size retention works only if you have data storage for at least 10% of your retention time.

And in my use case this last value is hard to compute in advance as my environment is dynamic.

I don't know the technical reason to do size retention cleanup only after a successful block compaction.

System information

No response

Prometheus version

prometheus, version 2.35.0 (branch: HEAD, revision: 6656cd29fe6ac92bab91ecec0fe162ef0f187654)
  build user:       root@cf6852b14d68
  build date:       20220421-09:53:42
  go version:       go1.18.1
  platform:         linux/amd64

Prometheus configuration file

--storage.tsdb.retention.time=10d --storage.tsdb.retention.size=3435973836B

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response