Basekick-Labs / Basekick-Labs/arc

WAL fills the default 10Gi PVC during bulk ingest: no backpressure, boot deadlock on full volume, liveness kills recovery

Open
#676 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

During sustained bulk ingest (snapshot backfill at ~100 MB/s), the write-ahead log can grow faster than it flushes. On a Kubernetes deployment with an S3/object-storage backend, the local PVC holds only the WAL and cache, and the chart's default `persistence.size: 10Gi` (`helm/arc/values.yaml`) fills in minutes at that rate. The resulting failure chain is worse than a full disk:

1. **WAL fills the volume with no backpressure.** Writes keep being accepted while the WAL grows unboundedly until the volume is full.
2. **Boot deadlock.** After the writer goes down, it cannot start again: creating the initial WAL file on startup requires free space on the already-full volume, so the pod can never come back without manual intervention (growing the PVC or deleting WAL segments by hand).
3. **Liveness kills recovery.** Once space is available, WAL replay after an unclean stop can exceed the liveness budget in `helm/arc/templates/deployment.yaml` (`periodSeconds: 10`, `failureThreshold: 3`, no `startupProbe`), so the pod is killed mid-recovery and crash-loops repeatedly before it manages to drain.

Observed in the field during a TB-scale migration; the operator's workaround was `ARC_WAL_ENABLED=false` for the migration window (throughput was unchanged, WAL stayed flat), which works but gives up durability for live writes during that window.

### Proposed fixes

- [ ] **Backpressure (core):** when WAL disk usage crosses a high-water mark, reject writes (503, or a typed backpressure error clients can retry) instead of growing until the volume is full. Reserve enough headroom that the writer can always create its initial WAL file on boot, so the deadlock in (2) becomes impossible.
- [ ] **`startupProbe` (chart):** add a `startupProbe` with a generous `failureThreshold × periodSeconds` window (minutes, not seconds) so WAL recovery is never killed by the liveness probe; keep the current livenessProbe for steady state.
- [ ] **Sizing guidance and defaults (chart/docs):** document WAL volume sizing for object-storage backends (roughly: peak ingest rate × worst-case flush lag, plus cache), call out that `10Gi` is a dev-scale default, and consider raising it or adding a loud comment in `values.yaml`.

### Related (probably a separate issue)

With very wide rows (tens of KB of line protocol per row), the ingest buffer's row-count bound (100k rows) and the WAL's payload byte cap (`WAL payload exceeds maximum allowed size`, 104,857,600 bytes) cannot be reconciled by tuning: a full flush of wide rows exceeds the cap, so those flushes never reach the WAL at all — the WAL directory holds only a 7-byte header file while ingest runs. Chunking WAL record writes, or bounding the ingest buffer by bytes as well as rows, would fix it. Happy to split this into its own issue.

Contributor guide

Open the contributing guide

Research direction

Start by tracing WAL space handling and recovery in the Go core, then inspect helm/arc/values.yaml and helm/arc/templates/deployment.yaml. Verify the proposed scope with the existing ingest and startup behavior. Done means writes apply bounded backpressure with boot headroom, the chart allows WAL recovery via a startupProbe, and WAL sizing guidance is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes
Domain
backend, cloud, devops, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.