Basekick-Labs / Basekick-Labs/arc
cluster: replicas register their replicated rows in the manifest — duplicate rows on query
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
A replica applies replicated rows into its own `ArrowBuffer`, and the cluster
file registrar is wired on every cluster node with no role gate. So the replica
flushes those rows to its *own* Parquet under its *own* key and registers that
file in the shared Raft manifest — while also pulling the primary's copy of the
same rows. Both copies are then queryable: duplicate rows.
## Where
`cmd/arc/main.go:1718-1720`, inside the cluster-coordinator block with no role
check:
```go
fileRegistrar := cluster.NewCoordinatorFileRegistrar(clusterCoordinator, logger.Get("file-registrar"))
fileRegistrar.Start(context.Background())
arrowBuffer.SetFileRegistrar(fileRegistrar)
```
`internal/ingest/arrow_writer.go:1255`:
```go
if b.fileRegistrar != nil {
b.fileRegistrar.RegisterFile(database, measurement, storagePath, partitionTime, sizeBytes, sha256Hex)
```
The replica's inbound path reaches the same buffer:
`Coordinator.buildReplicationIngestHandler` (`coordinator.go:3614`) calls
`buf.WriteColumnarDirectNoWAL(...)` (`arrow_writer.go:1487`). The `NoWAL` in the
name refers to skipping the WAL append, not to skipping the flush — the rows land
in the ordinary buffer and flush on the ordinary schedule.
## Consequence
In Pattern 1 with WAL replication actually running, for every replicated batch:
1. The primary flushes it to its local Parquet and registers it.
2. The replica applies the same rows, flushes them to its own local Parquet, and
registers that too.
3. Every node pulls both files.
Queries then see each replicated row twice.
## Why this is latent today
It needs a replica that is genuinely receiving live entries, which currently
almost never happens:
- Chart-deployed replicas never start a receiver at all (#886).
- Where one does start, it attaches to an arbitrary writer rather than the
primary, so roughly two in three receive nothing (#885).
Fixing either of those turns this on. It should be fixed **before** or
**with** them, not after.
## Suggested fix
Gate the registrar on the node's ingest capability, so only a node that
originates rows announces files. A replica's flushed Parquet is a local
materialisation of rows the primary already announced; it should not enter the
manifest.
Care is needed for the standalone/OSS path and for Pattern 2, where every writer
legitimately registers its own files — the gate wants to key off "did this node
originate these rows", not simply off role. The cleanest seam is probably to mark
replication-applied batches so their flush skips registration, rather than to
disable the registrar for the whole node, since a replica that is also a writer
in some configurations would otherwise stop announcing its own data.
Contributor guide
Research direction
Start at the cluster-coordinator block in cmd/arc/main.go:1718-1720, then trace registration in internal/ingest/arrow_writer.go:1255 and replicated ingestion through Coordinator.buildReplicationIngestHandler in coordinator.go:3614 and WriteColumnarDirectNoWAL at arrow_writer.go:1487. Verify that replication-applied batches do not enter the shared manifest, while standalone and Pattern 2 originating batches still register, and confirm queries no longer return duplicate rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100