Basekick-Labs / Basekick-Labs/arc

feat(ingest): add tag_keys support to MessagePack columnar format for auto-dedup

Open
#275 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Context

PR #274 adds automatic deduplication during compaction by storing tag column names as Parquet metadata (`arc:tags`). This works for Line Protocol and MessagePack row format, where tags and fields are distinct. However, the **MessagePack columnar format** sends a flat `columns` dict with no tag/field distinction, so files from this path don't get dedup metadata.

## Options

### Option A: Add `tag_keys` field to columnar payload (recommended)

```json
{m: "cpu", columns: {time: [...], host: [...], value: [...]}, tag_keys: ["host", "region"]}
```

- Sender declares which columns are tags
- Backwards compatible — if `tag_keys` is missing, no metadata written
- Minimal change: add `TagKeys []string` to `MsgPackPayload`, read in `decodeColumnar`

### Option B: Infer from column types

Tag columns are always strings, but field columns can also be strings (log messages, etc.). **Not viable** — can't reliably distinguish.

### Option C: Reuse `tags` dict with boolean markers

```json
{m: "cpu", columns: {...}, tags: {host: true, region: true}}
```

Overloads the existing `tags` field which has a different type in row format (`map[string]string`). Would require type-switching on the `tags` field, adding complexity.

## Recommendation

Option A is cleanest. One new field, explicit, backwards compatible, minimal code change.

## Files to modify

- `pkg/models/record.go` — add `TagKeys` to `MsgPackPayload`
- `internal/ingest/msgpack.go` — read `tag_keys` in `decodeColumnar`, set `ColumnarRecord.TagColumns`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.