Basekick-Labs / Basekick-Labs/arc
feat(ingest): add tag_keys support to MessagePack columnar format for auto-dedup
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Context
PR #274 adds automatic deduplication during compaction by storing tag column names as Parquet metadata (`arc:tags`). This works for Line Protocol and MessagePack row format, where tags and fields are distinct. However, the **MessagePack columnar format** sends a flat `columns` dict with no tag/field distinction, so files from this path don't get dedup metadata.
## Options
### Option A: Add `tag_keys` field to columnar payload (recommended)
```json
{m: "cpu", columns: {time: [...], host: [...], value: [...]}, tag_keys: ["host", "region"]}
```
- Sender declares which columns are tags
- Backwards compatible — if `tag_keys` is missing, no metadata written
- Minimal change: add `TagKeys []string` to `MsgPackPayload`, read in `decodeColumnar`
### Option B: Infer from column types
Tag columns are always strings, but field columns can also be strings (log messages, etc.). **Not viable** — can't reliably distinguish.
### Option C: Reuse `tags` dict with boolean markers
```json
{m: "cpu", columns: {...}, tags: {host: true, region: true}}
```
Overloads the existing `tags` field which has a different type in row format (`map[string]string`). Would require type-switching on the `tags` field, adding complexity.
## Recommendation
Option A is cleanest. One new field, explicit, backwards compatible, minimal code change.
## Files to modify
- `pkg/models/record.go` — add `TagKeys` to `MsgPackPayload`
- `internal/ingest/msgpack.go` — read `tag_keys` in `decodeColumnar`, set `ColumnarRecord.TagColumns`
Contributor guide
Assessment
This issue has not been assessed yet.