grafana / grafana/pyroscope

Disk fills up with old head blocks

Open
#4,418 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Describe the bug

We've noticed that blocks in the `head` directory sometimes don't get cleaned up. Eventually, there are so many old `head` blocks that the disk gets filled and we need to manually delete them.

The "cleanup on high disk utilization" settings described on https://grafana.com/docs/pyroscope/latest/configure-server/storage/configure-disk-storage/ only operate on blocks in the `local` directory, not the `head` directory, so changing those values didn't prevent our disk from filling up.

#### To Reproduce

We don't have a consistent method to reproduce it. We seem to be accruing old head blocks over time.

We sometimes see head blocks left dangling after we restart the pyroscope Pod, but we also see dangling head blocks created during time windows where our pyroscope Pod was running uninterrupted for weeks.

#### Expected behavior

We expect old blocks in `head` to eventually move to `local` and then get cleaned up.

#### Environment

- Infrastructure: Kubernetes on AWS
- Deployment tool: helm

We are running pyroscope in monolithic mode, via the helm chart + kustomize:
```yaml
---
apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
name: pyroscope

name: pyroscope
namespace: monitoring

repo: https://grafana.github.io/helm-charts
version: 1.13.1
releaseName: pyroscope

valuesInLine:
pyroscope:
replicaCount: 1

image:
pullPolicy: Always
repository: grafana/pyroscope
tag: 1.13.1

extraArgs:
config.expand-env: 'true'
distributor.ingestion-rate-limit-mb: "128"
ingester.max-global-series-per-tenant: "100000"
log.level: "info"
pyroscopedb.data-path: "/data"
pyroscopedb.retention-policy-min-disk-available-percentage: "0.05"
pyroscopedb.retention-policy-min-free-disk-gb: "10"
pyroscopedb.retention-policy-enforcement-interval: "5m"
pyroscopedb.max-block-duration: "1h"
pyroscopedb.retention-policy-disable: "false"
querier.max-query-length: "1d"
querier.max-query-lookback: "30d"

extraEnvFrom: [...]
extraEnvVars: [...]
extraVolumes: [...]
extraVolumeMounts: [...]

nameOverride: pyroscope

resources:
limits:
memory: [...]
requests:
cpu: [...]
memory: [...]

persistence:
enabled: true
size: 25Gi

terminationGracePeriodSeconds: 600

tolerations: [...]

structuredConfig:
storage:
storage_prefix: [...]
backend: [...]
s3: [...]

agent:
enabled: false
alloy:
enabled: false
```

#### Additional Context

Even when we turned on debug level logging, it didn't seem to log anything about blocks or errors relating to the ingestor or flushing. Are there any particular keywords I should search for in logs, or metrics to share that might be helpful in troubleshooting this?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.