Basekick-Labs / Basekick-Labs/arc
cluster: replicas never start a WAL replication receiver — the WAL gate is over-broad
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
In Pattern 1 (`cluster.shared_storage_mode=false`), replicas never start a WAL
replication receiver, so live-window replication is off in every chart-deployed
cluster. The gate that stops them is over-broad: it requires a local WAL, which
a *receiver* does not need.
## Where
`cmd/arc/main.go:1680`:
```go
if cfg.Cluster.ReplicationEnabled && walWriter != nil {
clusterCoordinator.SetWAL(walWriter)
clusterCoordinator.SetIngestBuffer(arrowBuffer)
if err := clusterCoordinator.StartReplication(); err != nil {
```
`walWriter` is non-nil only when `cfg.WAL.Enabled` (`cmd/arc/main.go:805`), which
defaults to `false` (`internal/config/config.go:1592`).
The enterprise chart sets `ARC_WAL_ENABLED` in exactly one helper,
`arc-enterprise.writerWalEnv` (`helm/arc-enterprise/templates/_helpers.tpl:325`),
included from exactly one place — `writer-statefulset.yaml:48`.
`reader-statefulset.yaml` and `compactor-statefulset.yaml` include
`commonClusterEnv`, `raftEnv` and `storageEnv`, none of which carry WAL env.
Replicas *do* get `ARC_CLUSTER_REPLICATION_ENABLED=true` in Pattern 1
(`_helpers.tpl:375`, the `storage.mode != shared` branch). The WAL conjunct is
the only thing stopping them.
## Consequence
On a chart-deployed Pattern 1 cluster:
- Readers and the compactor have `walWriter == nil`, so `StartReplication()` is
never called and no `Receiver` is ever constructed.
- Each writer still starts a `Sender` (`internal/cluster/coordinator.go:3477`)
that no one ever connects to.
- Replica freshness is whatever the Parquet manifest + peer-pull path gives —
roughly one flush interval (`ingest.max_buffer_age_ms`, default 5000ms) plus
pull latency — rather than the sub-second the WAL stream is for.
This also makes several replication defects dormant rather than fixed, including
#885.
## Why the gate is wrong, not the chart
A receiver does not require a local WAL. `Receiver.applyEntry`
(`internal/cluster/replication/receiver.go:695`) treats `LocalWAL` as optional:
```go
if r.cfg.LocalWAL != nil {
```
and applies to the ingest buffer via `IngestHandler` regardless. Only the
*sender* needs the WAL, because the WAL replication hook
(`coordinator.go:3497`) is what feeds `Sender.Replicate`.
So the fix is to split the gate — start replication whenever
`cluster.replication_enabled` is set, and require `walWriter` only for the
sender path — not to give every reader a WAL and a PVC.
## Suggested fix
Split the condition in `cmd/arc/main.go` so `StartReplication()` runs on
`cfg.Cluster.ReplicationEnabled` alone, with `SetWAL` called only when
`walWriter != nil`. `StartReplication` already branches on role
(`coordinator.go:3477`), so a writer without a WAL simply has nothing to send —
which should be logged as a misconfiguration rather than silently accepted.
No chart change is required.
Contributor guide
Research direction
Start at cmd/arc/main.go:1680 and inspect StartReplication's role branching in internal/cluster/coordinator.go:3477, then compare receiver handling in internal/cluster/replication/receiver.go:695. The change is complete when replication starts for configured replicas without a local WAL, while the sender still receives WAL only when available and the chart requires no changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, helm
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100