Basekick-Labs / Basekick-Labs/arc

cluster: line-protocol writes replicate into the `default` database, not the one they were written to

Open
#889 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Summary

Line-protocol writes replicate into the database `default` rather than the
database they were written to. The rows arrive on the replica intact, under the
wrong name, where queries against the real database never see them.

Native msgpack writes are unaffected — they carry the database in a WAL envelope
and land correctly.

## Mechanism

The WAL has two append paths and only one of them carries the database:

- `Writer.AppendRawWithMeta` (`internal/wal/wal.go:667`) frames the payload as
`[0x01 marker][2-byte db len][db name][msgpack]` and hands that enveloped
payload to the replication hook (`appendEnvelopedEntry`, `wal.go:708`).
- `Writer.Append` (`wal.go:657`) marshals the records and calls `AppendRaw`,
whose `appendRawEntry` (`wal.go:799`) fires the hook with the payload
**unenveloped**.

Line protocol always takes the second path. `ArrowBuffer.writeColumnarInternal`
(`internal/ingest/arrow_writer.go:1763`) uses the zero-copy
`AppendRawWithMeta` only when `record.RawPayload` is non-empty; a line-protocol
write has no raw msgpack, so it falls through to the branch commented
"FALLBACK: Convert columnar to row format for WAL storage / This path is used
for LineProtocol" and calls `b.wal.Append(walRecords)`.

On the receiving side, `Coordinator.buildReplicationIngestHandler`
(`internal/cluster/coordinator.go`) does:

```go
database, msgpackData := wal.ParseEnvelope(payload, "default")
```

`ParseEnvelope` (`wal.go:56`) returns the `defaultDB` argument when the marker
is absent, so `database` is `"default"`. The row-format branch then groups only
by measurement and writes every batch with that value.

The database is not actually lost in transit — `columnarToWALRecords`
(`arrow_writer.go`) stamps `"_database"` into every record — the handler just
never reads it.

## Reproduction

Pattern 1, three nodes (two writers, one reader), `wal.enabled=true` and
`cluster.replication_enabled=true` on all three, reader receiving from the
primary:

```
POST /api/v1/write/line-protocol (x-arc-database: rig) "live,host=h1 v=1 "
```

On the reader:

```
SELECT count(*) FROM default.live -- the replicated rows are here
SELECT count(*) FROM rig.live -- only what the file-pull path delivered
```

Observed on a live three-node rig: the reader's own storage tree held
`default/live/2026/09/16/17/live_*.parquet` alongside the pulled
`rig/live/.../live_*.parquet`.

## Fix

In the handler's row-format branch, group by `(database, measurement)` and take
the database from each record's `_database`, falling back to the envelope value
when the record carries no stamp.

## Sequencing — do not fix this alone

Fixing it in isolation makes observable behaviour worse. Today the misfiled rows
are invisible to queries against the real database, so the row count is right by
accident. Correcting the database puts them alongside the copy the file-pull
path already delivered, and the count doubles — #888, which a live rig confirms:
with the database corrected, a reader reported 6 rows where the primary reported
3, from two Parquet files in the same partition.

So this belongs with #888 (the replica must not materialise a second copy at
all) and #886 (the gate that turns replica replication on), as one piece of work
rather than three patches.

Contributor guide

Open the contributing guide

Research direction

Read internal/cluster/coordinator.go and wal.go first, then trace ArrowBuffer.writeColumnarInternal and columnarToWALRecords in internal/ingest/arrow_writer.go. Review #888 and #886 before starting, since this fix is explicitly sequenced with them. Done means replicated line-protocol rows use their stamped database without creating a duplicate copy.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.