Basekick-Labs / Basekick-Labs/arc
cluster: line-protocol writes replicate into the `default` database, not the one they were written to
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
Line-protocol writes replicate into the database `default` rather than the
database they were written to. The rows arrive on the replica intact, under the
wrong name, where queries against the real database never see them.
Native msgpack writes are unaffected — they carry the database in a WAL envelope
and land correctly.
## Mechanism
The WAL has two append paths and only one of them carries the database:
- `Writer.AppendRawWithMeta` (`internal/wal/wal.go:667`) frames the payload as
`[0x01 marker][2-byte db len][db name][msgpack]` and hands that enveloped
payload to the replication hook (`appendEnvelopedEntry`, `wal.go:708`).
- `Writer.Append` (`wal.go:657`) marshals the records and calls `AppendRaw`,
whose `appendRawEntry` (`wal.go:799`) fires the hook with the payload
**unenveloped**.
Line protocol always takes the second path. `ArrowBuffer.writeColumnarInternal`
(`internal/ingest/arrow_writer.go:1763`) uses the zero-copy
`AppendRawWithMeta` only when `record.RawPayload` is non-empty; a line-protocol
write has no raw msgpack, so it falls through to the branch commented
"FALLBACK: Convert columnar to row format for WAL storage / This path is used
for LineProtocol" and calls `b.wal.Append(walRecords)`.
On the receiving side, `Coordinator.buildReplicationIngestHandler`
(`internal/cluster/coordinator.go`) does:
```go
database, msgpackData := wal.ParseEnvelope(payload, "default")
```
`ParseEnvelope` (`wal.go:56`) returns the `defaultDB` argument when the marker
is absent, so `database` is `"default"`. The row-format branch then groups only
by measurement and writes every batch with that value.
The database is not actually lost in transit — `columnarToWALRecords`
(`arrow_writer.go`) stamps `"_database"` into every record — the handler just
never reads it.
## Reproduction
Pattern 1, three nodes (two writers, one reader), `wal.enabled=true` and
`cluster.replication_enabled=true` on all three, reader receiving from the
primary:
```
POST /api/v1/write/line-protocol (x-arc-database: rig) "live,host=h1 v=1 "
```
On the reader:
```
SELECT count(*) FROM default.live -- the replicated rows are here
SELECT count(*) FROM rig.live -- only what the file-pull path delivered
```
Observed on a live three-node rig: the reader's own storage tree held
`default/live/2026/09/16/17/live_*.parquet` alongside the pulled
`rig/live/.../live_*.parquet`.
## Fix
In the handler's row-format branch, group by `(database, measurement)` and take
the database from each record's `_database`, falling back to the envelope value
when the record carries no stamp.
## Sequencing — do not fix this alone
Fixing it in isolation makes observable behaviour worse. Today the misfiled rows
are invisible to queries against the real database, so the row count is right by
accident. Correcting the database puts them alongside the copy the file-pull
path already delivered, and the count doubles — #888, which a live rig confirms:
with the database corrected, a reader reported 6 rows where the primary reported
3, from two Parquet files in the same partition.
So this belongs with #888 (the replica must not materialise a second copy at
all) and #886 (the gate that turns replica replication on), as one piece of work
rather than three patches.
Contributor guide
Research direction
Read internal/cluster/coordinator.go and wal.go first, then trace ArrowBuffer.writeColumnarInternal and columnarToWALRecords in internal/ingest/arrow_writer.go. Review #888 and #886 before starting, since this fix is explicitly sequenced with them. Done means replicated line-protocol rows use their stamped database without creating a duplicate copy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100