canonical / canonical/observability-stack

docs: add steps for clearing out the WAL

Open
#302 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
9
Forks
12
Avg merge
2d 40m
Merged PRs (30d)
3

Description

Via @marino-mrc

otelcol gave:
```
Exporting failed. Dropping data.
...
Permanent error: Permanent error: context deadline exceeded
...
Permanent error: Permanent error: Permanent error: remote write returned HTTP status 429 Too Many Requests; the request has been rejected because the tenant exceeded the ingestion rate limit
```

On mimir-write side,
```
no space left on device
```

One option is to kill the pvc,
```bash
kubectl scale statefulset mimir-write -n cos --replicas=0
kubectl delete pvc mimir-write-data-db5430bc-mimir-write-0 mimir-write-data-db5430bc-mimir-write-1 mimir-write-data-db5430bc-mimir-write-2 mimir-write-recovery-data-db5430bc-mimir-write-0 mimir-write-recovery-data-db5430bc-mimir-write-1 mimir-write-recovery-data-db5430bc-mimir-write-2 -n cos
kubectl scale statefulset mimir-write -n cos --replicas=3
```

A quicker option is to `juju ssh` and delete the WAL files.

We should:
1. Add a doc showing how to clear out the WAL. Maybe even add charm action `purge-wal`.
2. Include the above error messages verbatim in our docs, so they are searchable.

---
*Reported from: https://documentation.ubuntu.com/observability/latest/how-to/deploy-and-manage/*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.