block / block/buzz

Chart mounts up to 17Gi of emptyDir per pod but sets no ephemeral-storage request

Open
#5,215 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

Component: deploy/charts/buzz (chart 0.1.7)
Ref: main
Severity: low — defaults/docs correctness, not a functional break

**Describe the bug**
In the chart's own recommended HA configuration, each relay pod mounts up to 17Gi
of sizeLimit-ed emptyDir, while the default relay.resources block declares no
ephemeral-storage request or limit. Because the kube-scheduler does not account for
emptyDir.sizeLimit, that disk footprint is invisible at scheduling time — pods co-pack
freely and are then evicted under node DiskPressure.

**Steps to reproduce**
1. Use the chart for an HA configuration

**Expected behavior**

Detail
values.yaml recommends disabling the git PVC for multi-replica HA:

```text
enabled: false → mount a per-pod emptyDir at mountPath (pure scratch) […] This is
the correct choice for multi-replica HA.
```

In that mode templates/deployment.yaml renders two emptyDir volumes:

```yaml
- name: git-repos
emptyDir:
sizeLimit: {{ .Values.persistence.git.size | quote }} # default 10Gi
- name: git-pack-cache
emptyDir:
sizeLimit: {{ .Values.git.packCacheVolumeSize | quote }} # default 7Gi
```
…for 17Gi per pod on defaults. But the default resources are:

```yaml
relay:
resources:
requests: { cpu: "500m", memory: "512Mi" }
limits: { cpu: "2", memory: "2Gi" }
```

No ephemeral-storage on either side.

_Why that matters_
emptyDir.sizeLimit is enforced by the kubelet eviction manager. Scheduling capacity is
computed only from resources.requests.ephemeral-storage. So the chart declares a
per-pod disk footprint the scheduler cannot see, with two consequences:

1. **Over-packing**. Nothing stops N relay pods (each entitled to 17Gi) landing on a node
with far less allocatable ephemeral storage.
2. **The relay is a first-choice eviction victim**. With no ephemeral-storage request,
all of the pod's ephemeral usage counts as above-request, which is how kubelet ranks
pods for ephemeral-storage eviction. Under DiskPressure the relay is evicted early —
and co-tenant pods can be taken with it.
This is most likely to bite exactly the users following the HA guidance, on
modest/default node disks, and it surfaces as unexplained pod evictions rather than
anything pointing at storage configuration.

**Version and platform**
Find your version at the bottom of the Settings sidebar. Write "unknown" if you can't determine it.
- Buzz version: unknown - helm chart 0.1.7
- OS: Linux

**Logs / additional context**
_Suggested fix_

Any of these, in rough order of preference:

1. **Add ephemeral-storage to the default relay.resources**, derived from the two
sizeLimit defaults plus a little overlay headroom (e.g. 20Gi). Self-consistent
defaults: what the chart mounts is what the chart requests.
2. **Document it** in values.yaml beside the persistence.git.enabled: false HA
guidance, and in the README's HA section — noting that operators must set
relay.resources.requests.ephemeral-storage to at least
persistence.git.size + git.packCacheVolumeSize for the scheduler to respect it.
3. **Optionally hard-fail or warn in _validate.tpl** when the emptyDir path is active
and no ephemeral-storage request is set — consistent with the chart's existing
fail-loud posture on Redis, relayUrl, and ownerPubkey.
No workaround is needed by operators today: relay.resources is rendered with toYaml,
so ephemeral-storage can already be supplied from values. This is purely about defaults
that match what the chart actually mounts.

_Note_
The related caps look internally coherent and are not part of this report —
git.packCacheMaxBytes (5GiB) sits inside git.packCacheVolumeSize (7Gi), which
values.yaml explains as leaving room for cold-population staging. Only the missing
scheduler-visible request seems off.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.