Basekick-Labs / Basekick-Labs/arc

cluster: replicas never start a WAL replication receiver — the WAL gate is over-broad

Open
#886 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Summary

In Pattern 1 (`cluster.shared_storage_mode=false`), replicas never start a WAL
replication receiver, so live-window replication is off in every chart-deployed
cluster. The gate that stops them is over-broad: it requires a local WAL, which
a *receiver* does not need.

## Where

`cmd/arc/main.go:1680`:

```go
if cfg.Cluster.ReplicationEnabled && walWriter != nil {
clusterCoordinator.SetWAL(walWriter)
clusterCoordinator.SetIngestBuffer(arrowBuffer)
if err := clusterCoordinator.StartReplication(); err != nil {
```

`walWriter` is non-nil only when `cfg.WAL.Enabled` (`cmd/arc/main.go:805`), which
defaults to `false` (`internal/config/config.go:1592`).

The enterprise chart sets `ARC_WAL_ENABLED` in exactly one helper,
`arc-enterprise.writerWalEnv` (`helm/arc-enterprise/templates/_helpers.tpl:325`),
included from exactly one place — `writer-statefulset.yaml:48`.
`reader-statefulset.yaml` and `compactor-statefulset.yaml` include
`commonClusterEnv`, `raftEnv` and `storageEnv`, none of which carry WAL env.

Replicas *do* get `ARC_CLUSTER_REPLICATION_ENABLED=true` in Pattern 1
(`_helpers.tpl:375`, the `storage.mode != shared` branch). The WAL conjunct is
the only thing stopping them.

## Consequence

On a chart-deployed Pattern 1 cluster:

- Readers and the compactor have `walWriter == nil`, so `StartReplication()` is
never called and no `Receiver` is ever constructed.
- Each writer still starts a `Sender` (`internal/cluster/coordinator.go:3477`)
that no one ever connects to.
- Replica freshness is whatever the Parquet manifest + peer-pull path gives —
roughly one flush interval (`ingest.max_buffer_age_ms`, default 5000ms) plus
pull latency — rather than the sub-second the WAL stream is for.

This also makes several replication defects dormant rather than fixed, including
#885.

## Why the gate is wrong, not the chart

A receiver does not require a local WAL. `Receiver.applyEntry`
(`internal/cluster/replication/receiver.go:695`) treats `LocalWAL` as optional:

```go
if r.cfg.LocalWAL != nil {
```

and applies to the ingest buffer via `IngestHandler` regardless. Only the
*sender* needs the WAL, because the WAL replication hook
(`coordinator.go:3497`) is what feeds `Sender.Replicate`.

So the fix is to split the gate — start replication whenever
`cluster.replication_enabled` is set, and require `walWriter` only for the
sender path — not to give every reader a WAL and a PVC.

## Suggested fix

Split the condition in `cmd/arc/main.go` so `StartReplication()` runs on
`cfg.Cluster.ReplicationEnabled` alone, with `SetWAL` called only when
`walWriter != nil`. `StartReplication` already branches on role
(`coordinator.go:3477`), so a writer without a WAL simply has nothing to send —
which should be logged as a misconfiguration rather than silently accepted.

No chart change is required.

Contributor guide

Open the contributing guide

Research direction

Start at cmd/arc/main.go:1680 and inspect StartReplication's role branching in internal/cluster/coordinator.go:3477, then compare receiver handling in internal/cluster/replication/receiver.go:695. The change is complete when replication starts for configured replicas without a local WAL, while the sender still receives WAL only when available and the chart requires no changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.