Disk fills up with old head blocks
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
#### Describe the bug
We've noticed that blocks in the `head` directory sometimes don't get cleaned up. Eventually, there are so many old `head` blocks that the disk gets filled and we need to manually delete them.
The "cleanup on high disk utilization" settings described on https://grafana.com/docs/pyroscope/latest/configure-server/storage/configure-disk-storage/ only operate on blocks in the `local` directory, not the `head` directory, so changing those values didn't prevent our disk from filling up.
#### To Reproduce
We don't have a consistent method to reproduce it. We seem to be accruing old head blocks over time.
We sometimes see head blocks left dangling after we restart the pyroscope Pod, but we also see dangling head blocks created during time windows where our pyroscope Pod was running uninterrupted for weeks.
#### Expected behavior
We expect old blocks in `head` to eventually move to `local` and then get cleaned up.
#### Environment
- Infrastructure: Kubernetes on AWS
- Deployment tool: helm
We are running pyroscope in monolithic mode, via the helm chart + kustomize:
```yaml
---
apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
name: pyroscope
name: pyroscope
namespace: monitoring
repo: https://grafana.github.io/helm-charts
version: 1.13.1
releaseName: pyroscope
valuesInLine:
pyroscope:
replicaCount: 1
image:
pullPolicy: Always
repository: grafana/pyroscope
tag: 1.13.1
extraArgs:
config.expand-env: 'true'
distributor.ingestion-rate-limit-mb: "128"
ingester.max-global-series-per-tenant: "100000"
log.level: "info"
pyroscopedb.data-path: "/data"
pyroscopedb.retention-policy-min-disk-available-percentage: "0.05"
pyroscopedb.retention-policy-min-free-disk-gb: "10"
pyroscopedb.retention-policy-enforcement-interval: "5m"
pyroscopedb.max-block-duration: "1h"
pyroscopedb.retention-policy-disable: "false"
querier.max-query-length: "1d"
querier.max-query-lookback: "30d"
extraEnvFrom: [...]
extraEnvVars: [...]
extraVolumes: [...]
extraVolumeMounts: [...]
nameOverride: pyroscope
resources:
limits:
memory: [...]
requests:
cpu: [...]
memory: [...]
persistence:
enabled: true
size: 25Gi
terminationGracePeriodSeconds: 600
tolerations: [...]
structuredConfig:
storage:
storage_prefix: [...]
backend: [...]
s3: [...]
agent:
enabled: false
alloy:
enabled: false
```
#### Additional Context
Even when we turned on debug level logging, it didn't seem to log anything about blocks or errors relating to the ingestor or flushing. Are there any particular keywords I should search for in logs, or metrics to share that might be helpful in troubleshooting this?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.